diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index cdf348b..2a12ab6 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -3042,3 +3042,124 @@ shared etcd/Longhorn device plus the I/O-pressure alert that missed the prod-1 o Branch pruning was deliberately NOT done: four agent branches read as "merged, no unique commits" only because their agents had not committed yet, and deleting those would have destroyed live work. + +## 2026-09-13 — qBittorrent minted a NEW admin password on every start; both arrs were already broken (branch `fix/qbittorrent-durable-webui-credential`) + +**THE DEFECT.** `/config/qBittorrent/qBittorrent.conf` on prod carried neither `WebUI\Username` nor +`WebUI\Password_PBKDF2`. qBittorrent 5.x reads that as "no credential configured" and generates a +fresh random `admin` password on EVERY process start, announcing it only on stdout. Two consequences, +and **both had already fired by the time this was finished** — this was not a hypothesis: + +- Sonarr's and Radarr's download-client credentials live in their own databases. Measured at 21:0xZ, + before any change: `POST /api/v3/downloadclient/testall` returned + `isValid: false, validationFailures: [{propertyName: "Username", errorMessage: "Authentication Failure"}]` + on **both** apps. Completed Download Handling was broken for both, silently — the arrs report a + failed client, not a failed import. +- No human had a stable login either; every restart meant reading the pod log. + +**THE MECHANISM.** A SOPS/age Secret `qbittorrent-bootstrap` (`webui-username`, `webui-password`, 32 +alphanumeric chars) is the single source of truth, consumed as env by the app container. A +`lifecycle.postStart` hook derives qBittorrent's own stored form — PBKDF2-HMAC-SHA512, 100000 +iterations, 16-byte random salt, 64-byte digest, stored as +`WebUI\Password_PBKDF2="@ByteArray(:)"` — asserts both keys into the conf on +**every** container start, and then **logs in to the WebUI to prove it works**. Authentication is +never weakened: `WebUI\AuthSubnetWhitelist` was rejected outright (it leaves the human login broken +and makes auth depend on network policy staying correct for ever). + +**HOW A REBUILD FROM GIT REPRODUCES IT.** All three paths converge on the same credential: +`pod restart` → the conf already holds it, the hook verifies and writes nothing (steady state does +not even rotate the salt); `config-PVC restored from an older snapshot` → a restore requires the pod +to be recreated, so the hook runs and re-asserts over whatever the snapshot held; `rebuild from +git onto an EMPTY PVC` → the image copies `/defaults/qBittorrent.conf`, the hook writes the +credential into it before the WebUI is serving. The only input is the SOPS file in git. + +**WHY postStart AND NOT THE REPO'S CronJob RECONCILER IDIOM** (`platform/keycloak/client-secret-reconciler.yaml`, +the CNPG grant reconciler in `apps/wave-0/nextcloud-db.yaml`). The idiom's *principle* — desired state +in git, RE-ASSERTED unconditionally, never set once — is exactly what is applied; only the trigger +differs, because the failure being closed IS a container start. qbittorrent-nox reads the credential +once, at process start, from a file on the PVC, so a 15-minute cron leaves a 0–15 minute window after +every restart in which the app serves a random password and both arrs are broken. A CronJob would also +have to reach the conf from OUTSIDE the pod (mounting an RWO Longhorn PVC the app already holds, or +`kubectl exec` with RBAC it would need granting) and then edit a file under a running process whose +own shutdown save can overwrite the edit. The hook needs no ServiceAccount, RBAC, NetworkPolicy or +API token: strictly less privilege. An **initContainer** was the genuinely tempting third option — it +would run strictly before the app and its stdout would reach `kubectl logs` — but it cannot perform +the ground-truth login (the WebUI is not up yet), and the ServiceDefinition schema 1.5.0 has +`lifecycle` but no initContainers field, so it would cost a schema bump plus a renderer change across +every service's golden to buy something strictly weaker. + +**TWO REAL BUGS FOUND IN THE INHERITED WIP, both from the postBuild-substitution trap.** The hook +computed its port as a doubled-dollar `WEBUI_PORT` default-operator expansion. That is *correct on the +cluster* (envsubst collapses the doubled dollar) and the renderer's `assert_substitution_safe` passes +it — but run any other way, bash reads the doubled dollar as its own PID, so `PORT` became +`{WEBUI_PORT:-8080}`, every curl got an unparseable URL, and the hook logged **"the WebUI +rejected the durable credential" three times having never asked it anything**. That misdiagnosis is +preserved in the service's own marker log at `2026-09-13T05:35Z`. Fixed by making it a literal +`PORT=8080` — the script now contains no braced dollar expansion at all and behaves identically +however it is run. Second bug: one shared wait budget covered both "wait for the s6 service" and +"wait for the API", so the first consumed all 90s and attempts 2 and 3 fired instantly. They now have +separate counters, "the API never answered" is reported as a distinct fault from "the password was +rejected", and a rejection now restarts qbittorrent-nox so a retry can actually change its outcome. + +**A SELF-INFLICTED IP BAN IS PART OF THE BLAST RADIUS, AND IT WILL FOOL AN OPERATOR.** Repointing the +arrs at the correct password FAILED at first: the arr's own save-time validation returned +`Authentication Failure`, and from inside the sonarr pod qBittorrent answered +`Your IP address has been banned after too many failed authentication attempts.` A day of arr retries +against the stale password had banned both arr pod IPs. The credential was right the whole time. The +ban is in-memory, so the qbittorrent **process restart cleared it**; nothing was relaxed +(`MaxAuthenticationFailCount`/`BanDuration` untouched — the fix is that the arrs stop generating +failures). The hook spends at most 3 logins per start for exactly this reason. + +**THE ARRS' STORED PASSWORD CANNOT BE READ BACK, so idempotency is driven by `testall`.** Sonarr and +Radarr return `"value": "********"` (a literal 8-asterisk mask, `privacy: password`) for every +password field. Comparing against that makes every run look like a change — and it also means the +earlier reading of "the arrs store an 8-char password" was reading the mask, not a value. The +convergence step therefore asks `POST /api/v3/downloadclient/testall` first and writes only when it +says `isValid: false`; a second run reports `ALREADY CONVERGED` and writes nothing. + +**PROOF ON PROD.** `testall` flipped `isValid: false → true` on both apps. Then the pod was deleted +and recreated (`...-v8rj4` → `...-gpjcz`, startTime `21:22:18Z`): the conf still held both keys, the +WebUI answered `Ok.` to a login with the credential decrypted from git, and **both** arrs still +reported `isValid: true` — so qBittorrent's shutdown rewrite of its own conf preserves the credential +rather than clobbering it. The post-envsubst manifest was audited as `flux build kustomization` +produces it: no doubled dollar, no braced expansion, `PORT=8080`, `bash -n` clean. The hook's write +path was exercised in the real container against a scratch copy of the conf under `/tmp`, using the +helper extracted verbatim from that post-envsubst output: exit 10 on a conf with the keys removed +(wrote them), exit 0 on a second run (converged, byte-identical file — no salt rotation), and the +digest it produced verifies under an independent PBKDF2 implementation with the git password and +fails with a wrong one. + +**WHAT IS *NOT* PROVEN, stated plainly.** The hook has **not** yet run as a real `postStart` on prod. +The Deployment is Flux-owned from `main`, and both ways of getting the branch's hook onto the live +object — suspending `apps-wave-0` and patching the Deployment — were refused by this session's +permission classifier. The durability shown above is therefore the *credential's* durability across +two real pod restarts, plus the hook's write/verify logic proven in-container; the kubelet actually +invoking it is verified only on merge. **First thing to check after merging:** that +`/config/qbittorrent-webui-cred.log` gains a `VERIFIED: the WebUI accepts the durable credential` +line on the next pod start. + +**STILL OUTSIDE GIT (unchanged by this, and it is app state by nature).** Sonarr's and Radarr's +download-client rows — including this password — live in their own databases on their config PVCs. +Rotating `webui-password` in the SOPS file therefore means re-running the convergence step against +both arrs; git alone will not do it. Same caveat the Remote Path Mappings entry already records. + +**The live-only Secret is now adopted.** `qbittorrent-bootstrap` existed on prod only as +`kubectl-client-side-apply`. Its SOPS file was in the WIP commit but **`apps/wave-0/kustomization.yaml` +did not list it**, so Flux would never have applied it and the adoption the last handoff expected +would silently not have happened. It is listed now (verified: the decrypted SOPS values are +byte-identical to the live Secret's, compared by SHA-256, so adoption cannot change the password). +Note also that the live Secret carries the plaintext in a +`kubectl.kubernetes.io/last-applied-configuration` annotation — a side effect of how it was created by +hand; Flux's server-side apply on merge does not write that annotation. + +**GATE: PASS** (`make -C ci gate`, exit 0, from a real clone — gating from inside a linked worktree +cannot work here because the container mounts only the worktree while its `.git` points outside). +spec-immutability, lint, l0 (13 cases, 0 failures — including `no-plaintext-secrets`), render-golden +(all 10 onboarded goldens reproduce from the canonical render) and l1 all green. The touched +requirement is **R-GITOPS-02**. Its L2 stage was then run deliberately against prod rather than left +skipped, and passed — **but that green is worth nothing and should not be cited as coverage**: the +only test covering R-GITOPS-02, `conformance/tests/l2-behavioral/onboarding-completeness`, is a +scaffold containing 8 `TODO` echoes and asserts none of the behaviours it names (compute up, PVC +bound, default-deny, egress class, DNS/TLS, metrics, update policy). This is a concrete instance of +the fail-open the `feat/coverage-debt-paydown` and `fix/touched-requirements-fail-closed` branches are +about: a stage that reports green while verifying nothing. diff --git a/apps/wave-0/kustomization.yaml b/apps/wave-0/kustomization.yaml index 4899214..aa3b049 100644 --- a/apps/wave-0/kustomization.yaml +++ b/apps/wave-0/kustomization.yaml @@ -17,6 +17,7 @@ resources: - pihole-oauth2-proxy-credentials.sops.yaml - pihole.yaml - plex.yaml +- qbittorrent-bootstrap-secret.sops.yaml - qbittorrent-oauth2-proxy-credentials.sops.yaml - qbittorrent.yaml - semaphore-db-app.sops.yaml diff --git a/apps/wave-0/qbittorrent.yaml b/apps/wave-0/qbittorrent.yaml index 80ec94c..da9e4d1 100644 --- a/apps/wave-0/qbittorrent.yaml +++ b/apps/wave-0/qbittorrent.yaml @@ -152,8 +152,9 @@ spec: command: - /bin/bash - -c - - "# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the\n# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.\n#\n# NOTE the `$${...}` escapes below \u2014 load-bearing. apps-wave-0 declares\n# postBuild.substituteFrom cluster-vars, so Flux runs envsubst over this whole manifest at\n# RECONCILE time, and a SINGLE-dollar brace expansion is not a cluster-var, so envsubst\n# would blank it. Bare `$VAR` and `$(cmd)` are left alone (see catalog/gitea.yaml for the\n# incident that taught this, and render.py's assert_substitution_safe, which enforces it).\nset -u\nCONF=/config/qBittorrent/qBittorrent.conf\nSVC=/run/service/svc-qbittorrent\nPORT=$${WEBUI_PORT:-8080}\n# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` \u2014 so a hook\n# that only echoes is effectively silent (gitea's hits the same wall and writes a marker\n# file for it). Every outcome therefore also lands, timestamped, on the persistent config\n# volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the\n# credential from git was asserted and accepted on this start.\nlog() {\n echo \"[qbt-webui-cred] $*\"\n printf '%s [qbt-webui-cred] %s\\n' \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\" \"$*\" \\\n >> /config/qbittorrent-webui-cred.log 2>/dev/null || true\n}\n\n# (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC\n# before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of\n# racing to create one it would then not overwrite.\nn=0\nwhile [ ! -s \"$CONF\" ] && [ \"$n\" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done\n\nwaited=0\nattempt=0\nwhile [ \"$attempt\" -lt 3 ]; do\n attempt=$((attempt + 1))\n\n # (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),\n # 10 = rewritten, anything else = could not write.\n python3 - \"$CONF\" <<'PY'\nimport base64, hashlib, hmac, os, stat, sys\n\nconf = sys.argv[1]\nuser = os.environ.get(\"QBT_WEBUI_USERNAME\", \"\")\npw = os.environ.get(\"QBT_WEBUI_PASSWORD\", \"\")\nif not user or not pw:\n sys.stderr.write(\"[qbt-webui-cred] credential env is empty - refusing to touch the conf\\n\")\n raise SystemExit(1)\n\n# qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000\n# iterations, 16-byte random salt, 64-byte digest, stored as\n# WebUI\\Password_PBKDF2=\"@ByteArray(:)\"\n# Get any one of them wrong and the app simply rejects every login.\nITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16\nUKEY, PKEY, SECTION = \"WebUI\\\\Username\", \"WebUI\\\\Password_PBKDF2\", \"[Preferences]\"\n\ndef matches(raw):\n \"\"\"True iff the stored digest is one OUR password produces (salt is read from it).\"\"\"\n v = raw.strip().strip('\"')\n if v.startswith(\"@ByteArray(\") and v.endswith(\")\"):\n v = v[len(\"@ByteArray(\"):-1]\n try:\n salt_b64, digest_b64 = v.split(\":\", 1)\n salt = base64.b64decode(salt_b64, validate=True)\n digest = base64.b64decode(digest_b64, validate=True)\n except Exception:\n return False\n if not salt or not digest:\n return False\n return hmac.compare_digest(\n hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, len(digest)), digest)\n\ntry:\n lines = open(conf, encoding=\"utf-8\", errors=\"replace\").read().splitlines()\nexcept FileNotFoundError:\n lines = []\n\ndef value_of(key):\n for i, line in enumerate(lines):\n if line.startswith(key + \"=\"):\n return i, line.split(\"=\", 1)[1]\n return -1, None\n\nui, uval = value_of(UKEY)\npi, pval = value_of(PKEY)\nif uval == user and pval is not None and matches(pval):\n raise SystemExit(0) # converged - leave the file (and its salt) alone\n\nsalt = os.urandom(SALT_LEN)\ndigest = hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, DIGEST_LEN)\nstored = base64.b64encode(salt).decode() + \":\" + base64.b64encode(digest).decode()\ndesired = [UKEY\ - \ + \"=\" + user, PKEY + '=\"@ByteArray(' + stored + ')\"']\n\nout = [line for i, line in enumerate(lines) if i not in (ui, pi)]\nif SECTION in out:\n at = out.index(SECTION) + 1\nelse:\n if out and out[-1].strip():\n out.append(\"\")\n out.append(SECTION)\n at = len(out)\nout[at:at] = desired\n\ntry:\n os.makedirs(os.path.dirname(conf) or \".\", exist_ok=True)\n tmp = conf + \".webui-cred.new\"\n with open(tmp, \"w\", encoding=\"utf-8\") as fh:\n fh.write(\"\\n\".join(out) + \"\\n\")\n fh.flush()\n os.fsync(fh.fileno())\n # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would\n # cost the app the ability to save its own settings, so carry the ownership over.\n try:\n st = os.stat(conf)\n os.chown(tmp, st.st_uid, st.st_gid)\n os.chmod(tmp, stat.S_IMODE(st.st_mode))\n except FileNotFoundError:\n os.chown(tmp, int(os.environ.get(\"PUID\", \"1000\")),\n int(os.environ.get(\"PGID\", \"1000\")))\n os.chmod(tmp, 0o664)\n os.replace(tmp, conf) # atomic: the app never reads a half-written conf\nexcept OSError as exc:\n sys.stderr.write(\"[qbt-webui-cred] cannot write %s: %s\\n\" % (conf, exc))\n raise SystemExit(1)\nraise SystemExit(10)\nPY\n rc=$?\n\n if [ \"$rc\" = 0 ]; then\n log \"conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)\"\n elif [ \"$rc\" = 10 ]; then\n log \"asserted WebUI Username + Password_PBKDF2 into $CONF\"\n # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY\n # have read the old, credential-less conf and be serving a random temporary password.\n # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.\n # Nothing to do when the service has not started yet - it will read the new file.\n while [ \"$waited\" -lt 90 ] && [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" != \"true\" ]; do\n waited=$((waited + 1)); sleep 1\n done\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"qbittorrent-nox was already running - restarting it to load the credential\"\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\"\n sleep 3\n fi\n else\n log \"FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep\" >&2\n log \"minting a random password every start and BOTH arrs will break. Fix and restart.\" >&2\n exit 0\n fi\n\n # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at all\n # (shared ~90s budget across attempts), then spend exactly ONE login per attempt -\n # at most 3 rejected logins, which stays under qBittorrent's default ban threshold.\n while [ \"$waited\" -lt 90 ]; do\n code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' \"http://127.0.0.1:$PORT/\" 2>/dev/null)\n [ -n \"$code\" ] && [ \"$code\" != \"000\" ] && break\n waited=$((waited + 1)); sleep 1\n done\n body=$(curl -sS -m 10 \\\n --data-urlencode \"username=$QBT_WEBUI_USERNAME\" \\\n --data-urlencode \"password=$QBT_WEBUI_PASSWORD\" \\\n \"http://127.0.0.1:$PORT/api/v2/auth/login\" 2>/dev/null)\n if [ \"$body\" = \"Ok.\" ]; then\n log \"VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'\"\n exit 0\n fi\n log \"WebUI rejected the durable credential (login said '$body') - reasserting\" >&2\ndone\n\nlog \"GAVE UP after $attempt attempts: the WebUI does not accept the credential from git.\" >&2\n# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.\n# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the\n# app, and the arrs' own download-client health check surfaces a broken credential.\nexit 0\n" + - "# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the\n# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.\n#\n# THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.\n# apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over\n# this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;\n# enforced by render.py's assert_substitution_safe) makes such a form correct ON THE\n# CLUSTER \u2014 but only there:\n# envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl\n# apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during\n# development) keeps `$$`, which bash expands to its own PID. That is not a hypothetical;\n# see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`\n# are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY\n# however it is run.\nset -u\nCONF=/config/qBittorrent/qBittorrent.conf\nSVC=/run/service/svc-qbittorrent\n# The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this\n# ServiceDefinition \u2014 which is the single place the renderer takes the containerPort, the\n# Service port and the HTTPRoute backend port from \u2014 and the image exposes no WEBUI_PORT\n# env var for a lookup to read anyway.\n# It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and\n# that cost real time: run without envsubst, bash read the doubled dollar as its own PID,\n# so PORT became \"{WEBUI_PORT:-8080}\" \u2014 the brace never expanded, the digits never\n# matched. Every curl below then got an unparseable URL, returned nothing, and the hook\n# logged \"the WebUI rejected the durable\n# credential\" \u2014 having never actually asked it. That MISDIAGNOSIS is preserved in this\n# service's own marker log at 2026-09-13T05:35Z (three \"rejected\" lines, then GAVE UP,\n# against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.\nPORT=8080\n# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` \u2014 so a hook\n# that only echoes is effectively silent (gitea's hits the same wall and writes a marker\n# file for it). Every outcome therefore also lands, timestamped, on the persistent config\n# volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the\n# credential from git was asserted and accepted on this start.\nlog() {\n echo \"[qbt-webui-cred] $*\"\n printf '%s [qbt-webui-cred] %s\\n' \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\" \"$*\" \\\n >> /config/qbittorrent-webui-cred.log 2>/dev/null || true\n}\n\n# (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC\n# before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of\n# racing to create one it would then not overwrite.\nn=0\nwhile [ ! -s \"$CONF\" ] && [ \"$n\" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done\n\nwaited=0\nattempt=0\nwhile [ \"$attempt\" -lt 3 ]; do\n attempt=$((attempt + 1))\n\n # (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),\n # 10 = rewritten, anything else = could not write.\n python3 - \"$CONF\" <<'PY'\nimport base64, hashlib, hmac, os, stat, sys\n\nconf = sys.argv[1]\nuser = os.environ.get(\"QBT_WEBUI_USERNAME\", \"\")\npw = os.environ.get(\"QBT_WEBUI_PASSWORD\", \"\")\nif not user or not pw:\n sys.stderr.write(\"[qbt-webui-cred] credential env is empty - refusing to touch the conf\\n\")\n raise SystemExit(1)\n\n# qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000\n# iterations, 16-byte random salt, 64-byte digest, stored as\n# WebUI\\Password_PBKDF2=\"@ByteArray(:)\"\n# Get any one of them wrong and the app simply rejects every login.\nITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16\nUKEY, PKEY, SECTION = \"WebUI\\\\Username\", \"WebUI\\\\Password_PBKDF2\"\ + , \"[Preferences]\"\n\ndef matches(raw):\n \"\"\"True iff the stored digest is one OUR password produces (salt is read from it).\"\"\"\n v = raw.strip().strip('\"')\n if v.startswith(\"@ByteArray(\") and v.endswith(\")\"):\n v = v[len(\"@ByteArray(\"):-1]\n try:\n salt_b64, digest_b64 = v.split(\":\", 1)\n salt = base64.b64decode(salt_b64, validate=True)\n digest = base64.b64decode(digest_b64, validate=True)\n except Exception:\n return False\n if not salt or not digest:\n return False\n return hmac.compare_digest(\n hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, len(digest)), digest)\n\ntry:\n lines = open(conf, encoding=\"utf-8\", errors=\"replace\").read().splitlines()\nexcept FileNotFoundError:\n lines = []\n\ndef value_of(key):\n for i, line in enumerate(lines):\n if line.startswith(key + \"=\"):\n return i, line.split(\"=\", 1)[1]\n return -1, None\n\nui, uval = value_of(UKEY)\npi, pval = value_of(PKEY)\nif uval == user and pval is not None and matches(pval):\n raise SystemExit(0) # converged - leave the file (and its salt) alone\n\nsalt = os.urandom(SALT_LEN)\ndigest = hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, DIGEST_LEN)\nstored = base64.b64encode(salt).decode() + \":\" + base64.b64encode(digest).decode()\ndesired = [UKEY + \"=\" + user, PKEY + '=\"@ByteArray(' + stored + ')\"']\n\nout = [line for i, line in enumerate(lines) if i not in (ui, pi)]\nif SECTION in out:\n at = out.index(SECTION) + 1\nelse:\n if out and out[-1].strip():\n out.append(\"\")\n out.append(SECTION)\n at = len(out)\nout[at:at] = desired\n\ntry:\n os.makedirs(os.path.dirname(conf) or \".\", exist_ok=True)\n tmp = conf + \".webui-cred.new\"\n with open(tmp, \"w\", encoding=\"utf-8\") as fh:\n fh.write(\"\\n\".join(out) + \"\\n\")\n fh.flush()\n os.fsync(fh.fileno())\n # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would\n # cost the app the ability to save its own settings, so carry the ownership over.\n try:\n st = os.stat(conf)\n os.chown(tmp, st.st_uid, st.st_gid)\n os.chmod(tmp, stat.S_IMODE(st.st_mode))\n except FileNotFoundError:\n os.chown(tmp, int(os.environ.get(\"PUID\", \"1000\")),\n int(os.environ.get(\"PGID\", \"1000\")))\n os.chmod(tmp, 0o664)\n os.replace(tmp, conf) # atomic: the app never reads a half-written conf\nexcept OSError as exc:\n sys.stderr.write(\"[qbt-webui-cred] cannot write %s: %s\\n\" % (conf, exc))\n raise SystemExit(1)\nraise SystemExit(10)\nPY\n rc=$?\n\n if [ \"$rc\" = 0 ]; then\n log \"conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)\"\n elif [ \"$rc\" = 10 ]; then\n log \"asserted WebUI Username + Password_PBKDF2 into $CONF\"\n # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY\n # have read the old, credential-less conf and be serving a random temporary password.\n # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.\n # Nothing to do when the service has not started yet - it will read the new file.\n # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the\n # 2026-09-13 failure spent the entire verification budget before the first login and\n # then fired attempts 2 and 3 with no wait left at all.\n sv=0\n while [ \"$sv\" -lt 60 ] && [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" != \"true\" ]; do\n sv=$((sv + 1)); sleep 1\n done\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"qbittorrent-nox was already running - restarting it to load the credential\"\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\"\n sleep 3\n fi\n else\n log \"FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep\" >&2\n log \"minting a random password\ + \ every start and BOTH arrs will break. Fix and restart.\" >&2\n exit 0\n fi\n\n # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL\n # (shared ~120s budget across attempts), then spend exactly ONE login per attempt -\n # at most 3 rejected logins, which stays under qBittorrent's default ban threshold.\n #\n # \"THE API NEVER ANSWERED\" IS A DIFFERENT FAULT FROM \"THE PASSWORD WAS REJECTED\", and\n # conflating them is what made this hook lie once already (see the PORT comment). An\n # unreachable WebUI is reported as unreachable, and the hook stops rather than spending\n # its remaining login attempts to produce two more identical wrong diagnoses.\n api_ok=0\n while [ \"$waited\" -lt 120 ]; do\n code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' \"http://127.0.0.1:$PORT/\" 2>/dev/null)\n if [ -n \"$code\" ] && [ \"$code\" != \"000\" ]; then api_ok=1; break; fi\n waited=$((waited + 1)); sleep 1\n done\n if [ \"$api_ok\" != 1 ]; then\n log \"the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the\" >&2\n log \"credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was\" >&2\n log \"never submitted. Do not rotate the credential on the strength of this line.\" >&2\n exit 0\n fi\n body=$(curl -sS -m 10 \\\n --data-urlencode \"username=$QBT_WEBUI_USERNAME\" \\\n --data-urlencode \"password=$QBT_WEBUI_PASSWORD\" \\\n \"http://127.0.0.1:$PORT/api/v2/auth/login\" 2>/dev/null)\n if [ \"$body\" = \"Ok.\" ]; then\n log \"VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'\"\n exit 0\n fi\n log \"WebUI rejected the durable credential (login said '$body')\" >&2\n # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean\n # the running qbittorrent-nox is serving something it read from an EARLIER conf, so\n # rewriting the same bytes on the next attempt would change nothing. The remedy is to\n # make the process RE-READ the file, so restart the supervised service and let the next\n # attempt re-verify. Without this the retry loop could never change its own outcome.\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"restarting qbittorrent-nox so it re-reads $CONF, then re-verifying\" >&2\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\" >&2\n sleep 3\n fi\ndone\n\nlog \"GAVE UP after $attempt attempts: the WebUI does not accept the credential from git.\" >&2\n# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.\n# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the\n# app, and the arrs' own download-client health check surfaces a broken credential.\nexit 0\n" - name: oauth2-proxy image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 imagePullPolicy: IfNotPresent diff --git a/catalog/qbittorrent.yaml b/catalog/qbittorrent.yaml index ae285da..840561f 100644 --- a/catalog/qbittorrent.yaml +++ b/catalog/qbittorrent.yaml @@ -87,6 +87,19 @@ lifecycle: # The gitea entry's postStart is the same pattern (idempotent bootstrap of a credential the app # keeps in its own state), and the schema documents this hook for exactly that use. # + # AND WHY NOT AN initContainer, which is the genuinely tempting third option: it would run strictly + # BEFORE qbittorrent-nox, removing the race the `s6-svc -r` below exists to cover, and its stdout + # would reach `kubectl logs -c ` instead of the kubelet. Two reasons it loses here: + # * It CANNOT do step (3) — the ground-truth login that proves the WebUI actually accepts the + # credential — because the WebUI is not running yet. That assertion is the most valuable line + # in this whole mechanism: it is what turns "we wrote a file" into "the credential works", and + # writing the right PBKDF2 bytes is precisely the part that is easy to get subtly wrong. + # * The ServiceDefinition schema (1.5.0) has `lifecycle` but NO initContainers field, so it would + # mean a schema version bump plus a renderer change — a blast radius across every service's + # golden — to buy something strictly weaker than what the hook already does. + # The honest summary: postStart trades a short, self-healing start-up race (detected and closed by + # the s6 restart + re-verify loop) for the ability to VERIFY. That is the right trade. + # # DURABILITY, the whole point — all three paths end with the SAME credential: # pod restart -> conf on the PVC already holds it; the hook verifies and writes NOTHING. # config-PVC restore from an older backup -> restore requires the pod to be recreated, so the @@ -107,15 +120,33 @@ lifecycle: # Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the # running WebUI accepts it. Idempotent: the steady-state path writes nothing at all. # - # NOTE the `$${...}` escapes below — load-bearing. apps-wave-0 declares - # postBuild.substituteFrom cluster-vars, so Flux runs envsubst over this whole manifest at - # RECONCILE time, and a SINGLE-dollar brace expansion is not a cluster-var, so envsubst - # would blank it. Bare `$VAR` and `$(cmd)` are left alone (see catalog/gitea.yaml for the - # incident that taught this, and render.py's assert_substitution_safe, which enforces it). + # THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE. + # apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over + # this whole manifest at RECONCILE time. The repo's escape contract (double the dollar; + # enforced by render.py's assert_substitution_safe) makes such a form correct ON THE + # CLUSTER — but only there: + # envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl + # apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during + # development) keeps `$$`, which bash expands to its own PID. That is not a hypothetical; + # see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)` + # are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY + # however it is run. set -u CONF=/config/qBittorrent/qBittorrent.conf SVC=/run/service/svc-qbittorrent - PORT=$${WEBUI_PORT:-8080} + # The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this + # ServiceDefinition — which is the single place the renderer takes the containerPort, the + # Service port and the HTTPRoute backend port from — and the image exposes no WEBUI_PORT + # env var for a lookup to read anyway. + # It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and + # that cost real time: run without envsubst, bash read the doubled dollar as its own PID, + # so PORT became "{WEBUI_PORT:-8080}" — the brace never expanded, the digits never + # matched. Every curl below then got an unparseable URL, returned nothing, and the hook + # logged "the WebUI rejected the durable + # credential" — having never actually asked it. That MISDIAGNOSIS is preserved in this + # service's own marker log at 2026-09-13T05:35Z (three "rejected" lines, then GAVE UP, + # against a WebUI that accepted the credential perfectly well). A literal cannot be mangled. + PORT=8080 # A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` — so a hook # that only echoes is effectively silent (gitea's hits the same wall and writes a marker # file for it). Every outcome therefore also lands, timestamped, on the persistent config @@ -237,8 +268,12 @@ lifecycle: # have read the old, credential-less conf and be serving a random temporary password. # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written. # Nothing to do when the service has not started yet - it will read the new file. - while [ "$waited" -lt 90 ] && [ "$(s6-svstat -u "$SVC" 2>/dev/null)" != "true" ]; do - waited=$((waited + 1)); sleep 1 + # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the + # 2026-09-13 failure spent the entire verification budget before the first login and + # then fired attempts 2 and 3 with no wait left at all. + sv=0 + while [ "$sv" -lt 60 ] && [ "$(s6-svstat -u "$SVC" 2>/dev/null)" != "true" ]; do + sv=$((sv + 1)); sleep 1 done if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then log "qbittorrent-nox was already running - restarting it to load the credential" @@ -251,14 +286,26 @@ lifecycle: exit 0 fi - # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at all - # (shared ~90s budget across attempts), then spend exactly ONE login per attempt - + # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL + # (shared ~120s budget across attempts), then spend exactly ONE login per attempt - # at most 3 rejected logins, which stays under qBittorrent's default ban threshold. - while [ "$waited" -lt 90 ]; do + # + # "THE API NEVER ANSWERED" IS A DIFFERENT FAULT FROM "THE PASSWORD WAS REJECTED", and + # conflating them is what made this hook lie once already (see the PORT comment). An + # unreachable WebUI is reported as unreachable, and the hook stops rather than spending + # its remaining login attempts to produce two more identical wrong diagnoses. + api_ok=0 + while [ "$waited" -lt 120 ]; do code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' "http://127.0.0.1:$PORT/" 2>/dev/null) - [ -n "$code" ] && [ "$code" != "000" ] && break + if [ -n "$code" ] && [ "$code" != "000" ]; then api_ok=1; break; fi waited=$((waited + 1)); sleep 1 done + if [ "$api_ok" != 1 ]; then + log "the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the" >&2 + log "credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was" >&2 + log "never submitted. Do not rotate the credential on the strength of this line." >&2 + exit 0 + fi body=$(curl -sS -m 10 \ --data-urlencode "username=$QBT_WEBUI_USERNAME" \ --data-urlencode "password=$QBT_WEBUI_PASSWORD" \ @@ -267,7 +314,17 @@ lifecycle: log "VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'" exit 0 fi - log "WebUI rejected the durable credential (login said '$body') - reasserting" >&2 + log "WebUI rejected the durable credential (login said '$body')" >&2 + # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean + # the running qbittorrent-nox is serving something it read from an EARLIER conf, so + # rewriting the same bytes on the next attempt would change nothing. The remedy is to + # make the process RE-READ the file, so restart the supervised service and let the next + # attempt re-verify. Without this the retry loop could never change its own outcome. + if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then + log "restarting qbittorrent-nox so it re-reads $CONF, then re-verifying" >&2 + s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)" >&2 + sleep 3 + fi done log "GAVE UP after $attempt attempts: the WebUI does not accept the credential from git." >&2