From 204f0ec28e719bf59ee299f15a5fccfb2ddb1bb1 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 13 Sep 2026 08:52:21 -0500 Subject: [PATCH 1/2] WIP: durable qBittorrent WebUI credential MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit INCOMPLETE — preserved by the PM after the authoring agent hit the session rate limit mid-task. Do not merge without finishing and gating it. Problem: qBittorrent 5.1.2 has no WebUI\Username or WebUI\Password_PBKDF2 in its config, so it mints a NEW temporary admin password on every process start and announces it only on stdout. Two consequences, both real: the Sonarr and Radarr download clients configured earlier will silently stop working at the next pod restart, and a human operator has no stable login either. Adds a SOPS-encrypted qbittorrent-bootstrap Secret (verified encrypted: both values are ENC[AES256_GCM]) plus the catalog/rendered wiring to consume it. WHAT REMAINS: - The agent's last note: the postStart hook works, but its output never reaches kubectl logs, so it was switching to the marker-file logging pattern gitea's hook already uses. That change is NOT here. - The Secret exists on prod as kubectl-client-side-apply, i.e. live-only; merging adopts it into git. - Sonarr's and Radarr's stored download-client passwords were to be updated to the durable value — NOT done, so the chain may break on restart until it is. - The required proof is unfinished: restart the pod, then show the WebUI accepts the credential AND both arrs still report isValid: true. Co-Authored-By: Claude Opus 5 --- .../qbittorrent-bootstrap-secret.sops.yaml | 60 +++++ apps/wave-0/qbittorrent.yaml | 18 ++ catalog/qbittorrent.yaml | 232 ++++++++++++++++++ 3 files changed, 310 insertions(+) create mode 100644 apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml diff --git a/apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml b/apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml new file mode 100644 index 0000000..126ce16 --- /dev/null +++ b/apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml @@ -0,0 +1,60 @@ +# qBittorrent DURABLE WebUI credential (WS-APPS). SOPS-encrypted (age), decrypted in-cluster by Flux. +# +# WHY THIS FILE EXISTS (defect, measured on prod 2026-09-13): +# qBittorrent.conf on the config PVC carried NEITHER `WebUI\Username` NOR +# `WebUI\Password_PBKDF2`. qBittorrent 5.x treats that as "no credential set" and generates a NEW +# random `admin` password on EVERY process start, announcing it ONLY on stdout ("A temporary +# password is provided for this session: ..."). Consequences, both real: (a) Sonarr's and Radarr's +# saved download-client credentials stop working at the next qbittorrent pod restart, silently +# breaking Completed Download Handling; (b) a human operator has no stable login at all and must +# read the pod log after every restart. +# +# HOW IT IS APPLIED: catalog/qbittorrent.yaml's app container reads BOTH keys below as env +# (QBT_WEBUI_USERNAME / QBT_WEBUI_PASSWORD) and its `lifecycle.postStart` hook — the same idiom as +# gitea's break-glass admin bootstrap — DERIVES the PBKDF2-HMAC-SHA512 digest qBittorrent stores +# and asserts both keys into /config/qBittorrent/qBittorrent.conf on EVERY container start, then +# verifies the WebUI actually accepts them. Level-triggered: a pod restart, a config-PVC restore +# from an older backup, and a rebuild-from-git onto an EMPTY PVC all end with this SAME +# credential, because the hook re-asserts it on every start rather than once. +# +# webui-username: the WebUI account name. qBittorrent has exactly ONE local account, and `admin` is +# the name its own temporary-password path uses, so keeping it avoids inventing a second identity. +# webui-password: the durable plaintext. It is ALSO what Sonarr's and Radarr's download-client +# entries store in their own databases (app state on their config PVCs, NOT in git — see +# DECISIONS-LOG), so rotating it here means re-pointing both arrs. +# Deliberately alphanumeric: it travels through a urlencoded WebAPI login body and two arr JSON +# payloads, and a shell/URL-special character in any of those is a bug waiting to happen. +# +# NEVER put a value in a comment: SOPS does NOT encrypt comments. +# Recover with: infra/devcontainer/dev.sh run sops -d apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml +apiVersion: v1 +kind: Secret +metadata: + name: qbittorrent-bootstrap + namespace: qbittorrent + labels: + app.kubernetes.io/name: qbittorrent +type: Opaque +stringData: + webui-username: ENC[AES256_GCM,data:P022nG0=,iv:MXcCXsD5hs7MAJ+OAYHkZ3pXep7XGPqeAExChoqna08=,tag:WE34mxtfQRgiwPRaDA12pQ==,type:str] + webui-password: ENC[AES256_GCM,data:+Lx0Ny7SBZeF/CzQYm8umntw27ewyXS7unWObpl0fCg=,iv:JkAtublBMS7SJUutAFip1XjdmsKnGe+S2E9QDT4rCJk=,tag:MYptPVgoEUEpXe5kmg1xIg==,type:str] +sops: + kms: [] + gcp_kms: [] + azure_kv: [] + hc_vault: [] + age: + - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg + enc: | + -----BEGIN AGE ENCRYPTED FILE----- + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB1aWpUY04wZHoyWVdPbGU2 + TC9vaU5oYmhWSVh3YWVTdzkvU3Axdlh6QVFZCkV4MldaZXNyam5CejQyZStyczJ3 + UStFd0hRQmRsNGZLVEVTRnVjWXp6SVkKLS0tICtkWlRqVGN5OWYvVnZUaHYvbEZv + VWtHMVdDU3hmdXo1TndLb05zQzJ0S28KfqlDBTocO0ZAKtH6ad/00EeVRqZ7WMwv + FP2evd5vyn95QQICWhY/LH6EKbwVVlcIlL7i9l8K3kK4Ebdk54mAyw== + -----END AGE ENCRYPTED FILE----- + lastmodified: "2026-09-13T05:23:23Z" + mac: ENC[AES256_GCM,data:hQ7jIgFqpNcFY5H4AfMxuslC2R/2wGmGWtghemOmTUUipSdoGdbNjWGQhxzk40lFjw6Zj2b+buOpXpg70s4xr+4feusqa2TJ3+bxAuRWmuZpTQAU2OaO5v0SdDEs3gwKm2mA3JFo/e6POm/P/kf2GIECcyU1AciZQt6ws7UJmQc=,iv:6SF0ZAuzHQThD/FGbWhFfa4dXU8hyW8jWFiDmwgfFU8=,tag:31i8HM2s/mGeS4zvFxO53Q==,type:str] + pgp: [] + encrypted_regex: ^(data|stringData)$ + version: 3.9.4 diff --git a/apps/wave-0/qbittorrent.yaml b/apps/wave-0/qbittorrent.yaml index 25ede18..80ec94c 100644 --- a/apps/wave-0/qbittorrent.yaml +++ b/apps/wave-0/qbittorrent.yaml @@ -130,12 +130,30 @@ spec: value: '1000' - name: PGID value: '3000' + - name: QBT_WEBUI_USERNAME + valueFrom: + secretKeyRef: + name: qbittorrent-bootstrap + key: webui-username + - name: QBT_WEBUI_PASSWORD + valueFrom: + secretKeyRef: + name: qbittorrent-bootstrap + key: webui-password volumeMounts: - name: config mountPath: /config - name: downloads mountPath: /downloads subPath: downloads + lifecycle: + postStart: + exec: + command: + - /bin/bash + - -c + - "# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the\n# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.\n#\n# NOTE the `$${...}` escapes below \u2014 load-bearing. apps-wave-0 declares\n# postBuild.substituteFrom cluster-vars, so Flux runs envsubst over this whole manifest at\n# RECONCILE time, and a SINGLE-dollar brace expansion is not a cluster-var, so envsubst\n# would blank it. Bare `$VAR` and `$(cmd)` are left alone (see catalog/gitea.yaml for the\n# incident that taught this, and render.py's assert_substitution_safe, which enforces it).\nset -u\nCONF=/config/qBittorrent/qBittorrent.conf\nSVC=/run/service/svc-qbittorrent\nPORT=$${WEBUI_PORT:-8080}\n# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` \u2014 so a hook\n# that only echoes is effectively silent (gitea's hits the same wall and writes a marker\n# file for it). Every outcome therefore also lands, timestamped, on the persistent config\n# volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the\n# credential from git was asserted and accepted on this start.\nlog() {\n echo \"[qbt-webui-cred] $*\"\n printf '%s [qbt-webui-cred] %s\\n' \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\" \"$*\" \\\n >> /config/qbittorrent-webui-cred.log 2>/dev/null || true\n}\n\n# (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC\n# before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of\n# racing to create one it would then not overwrite.\nn=0\nwhile [ ! -s \"$CONF\" ] && [ \"$n\" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done\n\nwaited=0\nattempt=0\nwhile [ \"$attempt\" -lt 3 ]; do\n attempt=$((attempt + 1))\n\n # (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),\n # 10 = rewritten, anything else = could not write.\n python3 - \"$CONF\" <<'PY'\nimport base64, hashlib, hmac, os, stat, sys\n\nconf = sys.argv[1]\nuser = os.environ.get(\"QBT_WEBUI_USERNAME\", \"\")\npw = os.environ.get(\"QBT_WEBUI_PASSWORD\", \"\")\nif not user or not pw:\n sys.stderr.write(\"[qbt-webui-cred] credential env is empty - refusing to touch the conf\\n\")\n raise SystemExit(1)\n\n# qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000\n# iterations, 16-byte random salt, 64-byte digest, stored as\n# WebUI\\Password_PBKDF2=\"@ByteArray(:)\"\n# Get any one of them wrong and the app simply rejects every login.\nITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16\nUKEY, PKEY, SECTION = \"WebUI\\\\Username\", \"WebUI\\\\Password_PBKDF2\", \"[Preferences]\"\n\ndef matches(raw):\n \"\"\"True iff the stored digest is one OUR password produces (salt is read from it).\"\"\"\n v = raw.strip().strip('\"')\n if v.startswith(\"@ByteArray(\") and v.endswith(\")\"):\n v = v[len(\"@ByteArray(\"):-1]\n try:\n salt_b64, digest_b64 = v.split(\":\", 1)\n salt = base64.b64decode(salt_b64, validate=True)\n digest = base64.b64decode(digest_b64, validate=True)\n except Exception:\n return False\n if not salt or not digest:\n return False\n return hmac.compare_digest(\n hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, len(digest)), digest)\n\ntry:\n lines = open(conf, encoding=\"utf-8\", errors=\"replace\").read().splitlines()\nexcept FileNotFoundError:\n lines = []\n\ndef value_of(key):\n for i, line in enumerate(lines):\n if line.startswith(key + \"=\"):\n return i, line.split(\"=\", 1)[1]\n return -1, None\n\nui, uval = value_of(UKEY)\npi, pval = value_of(PKEY)\nif uval == user and pval is not None and matches(pval):\n raise SystemExit(0) # converged - leave the file (and its salt) alone\n\nsalt = os.urandom(SALT_LEN)\ndigest = hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, DIGEST_LEN)\nstored = base64.b64encode(salt).decode() + \":\" + base64.b64encode(digest).decode()\ndesired = [UKEY\ + \ + \"=\" + user, PKEY + '=\"@ByteArray(' + stored + ')\"']\n\nout = [line for i, line in enumerate(lines) if i not in (ui, pi)]\nif SECTION in out:\n at = out.index(SECTION) + 1\nelse:\n if out and out[-1].strip():\n out.append(\"\")\n out.append(SECTION)\n at = len(out)\nout[at:at] = desired\n\ntry:\n os.makedirs(os.path.dirname(conf) or \".\", exist_ok=True)\n tmp = conf + \".webui-cred.new\"\n with open(tmp, \"w\", encoding=\"utf-8\") as fh:\n fh.write(\"\\n\".join(out) + \"\\n\")\n fh.flush()\n os.fsync(fh.fileno())\n # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would\n # cost the app the ability to save its own settings, so carry the ownership over.\n try:\n st = os.stat(conf)\n os.chown(tmp, st.st_uid, st.st_gid)\n os.chmod(tmp, stat.S_IMODE(st.st_mode))\n except FileNotFoundError:\n os.chown(tmp, int(os.environ.get(\"PUID\", \"1000\")),\n int(os.environ.get(\"PGID\", \"1000\")))\n os.chmod(tmp, 0o664)\n os.replace(tmp, conf) # atomic: the app never reads a half-written conf\nexcept OSError as exc:\n sys.stderr.write(\"[qbt-webui-cred] cannot write %s: %s\\n\" % (conf, exc))\n raise SystemExit(1)\nraise SystemExit(10)\nPY\n rc=$?\n\n if [ \"$rc\" = 0 ]; then\n log \"conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)\"\n elif [ \"$rc\" = 10 ]; then\n log \"asserted WebUI Username + Password_PBKDF2 into $CONF\"\n # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY\n # have read the old, credential-less conf and be serving a random temporary password.\n # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.\n # Nothing to do when the service has not started yet - it will read the new file.\n while [ \"$waited\" -lt 90 ] && [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" != \"true\" ]; do\n waited=$((waited + 1)); sleep 1\n done\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"qbittorrent-nox was already running - restarting it to load the credential\"\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\"\n sleep 3\n fi\n else\n log \"FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep\" >&2\n log \"minting a random password every start and BOTH arrs will break. Fix and restart.\" >&2\n exit 0\n fi\n\n # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at all\n # (shared ~90s budget across attempts), then spend exactly ONE login per attempt -\n # at most 3 rejected logins, which stays under qBittorrent's default ban threshold.\n while [ \"$waited\" -lt 90 ]; do\n code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' \"http://127.0.0.1:$PORT/\" 2>/dev/null)\n [ -n \"$code\" ] && [ \"$code\" != \"000\" ] && break\n waited=$((waited + 1)); sleep 1\n done\n body=$(curl -sS -m 10 \\\n --data-urlencode \"username=$QBT_WEBUI_USERNAME\" \\\n --data-urlencode \"password=$QBT_WEBUI_PASSWORD\" \\\n \"http://127.0.0.1:$PORT/api/v2/auth/login\" 2>/dev/null)\n if [ \"$body\" = \"Ok.\" ]; then\n log \"VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'\"\n exit 0\n fi\n log \"WebUI rejected the durable credential (login said '$body') - reasserting\" >&2\ndone\n\nlog \"GAVE UP after $attempt attempts: the WebUI does not accept the credential from git.\" >&2\n# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.\n# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the\n# app, and the arrs' own download-client health check surfaces a broken credential.\nexit 0\n" - name: oauth2-proxy image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 imagePullPolicy: IfNotPresent diff --git a/catalog/qbittorrent.yaml b/catalog/qbittorrent.yaml index fc0393e..ae285da 100644 --- a/catalog/qbittorrent.yaml +++ b/catalog/qbittorrent.yaml @@ -44,6 +44,238 @@ env: value: "3000" # the export's dominant group, matching sonarr/radarr: files this writes stay # readable to the EXISTING non-k3s consumers of the same dataset, and land # group-owned the same way the arrs' own writes do + # --- THE DURABLE WebUI CREDENTIAL (consumed by the postStart hook below) ------------------- + # Measured on prod 2026-09-13: /config/qBittorrent/qBittorrent.conf held NEITHER `WebUI\Username` + # NOR `WebUI\Password_PBKDF2`. qBittorrent 5.x reads that as "no credential configured" and mints a + # NEW random `admin` password on EVERY process start, announcing it ONLY on stdout ("A temporary + # password is provided for this session: ..."). Two real consequences, not hypotheticals: + # * Sonarr and Radarr store a download-client username/password in their OWN databases. Both + # tested valid against the then-current temporary password; both would have started failing at + # the NEXT qbittorrent pod restart, silently breaking Completed Download Handling (the arrs + # report a failed client, not a failed import). + # * The human operator had no stable login: every restart meant reading the pod log. + # Provided as apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml (SOPS/age, decrypted by Flux) — + # same shape and naming as gitea-bootstrap, so `render.py --emit-waves` places it in this wave. + - name: QBT_WEBUI_USERNAME + valueFrom: + secretKeyRef: + name: qbittorrent-bootstrap + key: webui-username + - name: QBT_WEBUI_PASSWORD + valueFrom: + secretKeyRef: + name: qbittorrent-bootstrap + key: webui-password +lifecycle: + # LEVEL-TRIGGERED CREDENTIAL ASSERTION, run on EVERY container start. + # + # WHY A postStart HOOK AND NOT A CronJob RECONCILER: the repo's idiom for "state Flux cannot reach + # directly" is the level-triggered CronJob (platform/keycloak/client-secret-reconciler.yaml, + # apps/wave-0/nextcloud-db.yaml). That idiom's PRINCIPLE — desired state lives in git and is + # RE-ASSERTED unconditionally, never set once — is exactly what is applied here, but the trigger is + # the container start rather than a clock, because the failure this closes IS a container start: + # * The credential is read by qbittorrent-nox ONCE, at process start, from a file on the config + # PVC. A 15-minute cron would therefore leave a 0-15 minute window after every restart, PVC + # restore or rebuild in which the app serves a random password and both arrs are broken. The + # hook has no such window: it runs before/with the very start that would otherwise regenerate. + # * A CronJob would also have to reach the conf from OUTSIDE the pod — mounting an RWO Longhorn + # PVC that the app already holds, or `kubectl exec`-ing with RBAC it would need granting — and + # then edit a file UNDER the running process, whose own save-on-exit can overwrite the edit. + # In-container, before the service starts, none of that exists. + # * It needs no ServiceAccount, no RBAC, no NetworkPolicy and no API token: strictly less + # privilege than the CronJob alternative (R-SEC/least-privilege). + # The gitea entry's postStart is the same pattern (idempotent bootstrap of a credential the app + # keeps in its own state), and the schema documents this hook for exactly that use. + # + # DURABILITY, the whole point — all three paths end with the SAME credential: + # pod restart -> conf on the PVC already holds it; the hook verifies and writes NOTHING. + # config-PVC restore from an older backup -> restore requires the pod to be recreated, so the + # hook runs and re-asserts over whatever the snapshot contained. + # rebuild from git -> empty PVC; the image copies /defaults/qBittorrent.conf, the hook writes the + # credential into it before the service is serving. Nothing outside git. + # + # WHAT IT DOES NOT DO: it never weakens authentication. `WebUI\AuthSubnetWhitelist` (bypassing auth + # for the pod CIDR) was REJECTED: it leaves the human login broken, and makes qBittorrent's auth + # depend on network policy staying correct forever. The WebUI keeps demanding a password; that + # password is simply now known, fixed, and reproducible from git. + postStart: + exec: + command: + - /bin/bash + - -c + - | + # Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the + # running WebUI accepts it. Idempotent: the steady-state path writes nothing at all. + # + # NOTE the `$${...}` escapes below — load-bearing. apps-wave-0 declares + # postBuild.substituteFrom cluster-vars, so Flux runs envsubst over this whole manifest at + # RECONCILE time, and a SINGLE-dollar brace expansion is not a cluster-var, so envsubst + # would blank it. Bare `$VAR` and `$(cmd)` are left alone (see catalog/gitea.yaml for the + # incident that taught this, and render.py's assert_substitution_safe, which enforces it). + set -u + CONF=/config/qBittorrent/qBittorrent.conf + SVC=/run/service/svc-qbittorrent + PORT=$${WEBUI_PORT:-8080} + # A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` — so a hook + # that only echoes is effectively silent (gitea's hits the same wall and writes a marker + # file for it). Every outcome therefore also lands, timestamped, on the persistent config + # volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the + # credential from git was asserted and accepted on this start. + log() { + echo "[qbt-webui-cred] $*" + printf '%s [qbt-webui-cred] %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" \ + >> /config/qbittorrent-webui-cred.log 2>/dev/null || true + } + + # (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC + # before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of + # racing to create one it would then not overwrite. + n=0 + while [ ! -s "$CONF" ] && [ "$n" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done + + waited=0 + attempt=0 + while [ "$attempt" -lt 3 ]; do + attempt=$((attempt + 1)) + + # (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt), + # 10 = rewritten, anything else = could not write. + python3 - "$CONF" <<'PY' + import base64, hashlib, hmac, os, stat, sys + + conf = sys.argv[1] + user = os.environ.get("QBT_WEBUI_USERNAME", "") + pw = os.environ.get("QBT_WEBUI_PASSWORD", "") + if not user or not pw: + sys.stderr.write("[qbt-webui-cred] credential env is empty - refusing to touch the conf\n") + raise SystemExit(1) + + # qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000 + # iterations, 16-byte random salt, 64-byte digest, stored as + # WebUI\Password_PBKDF2="@ByteArray(:)" + # Get any one of them wrong and the app simply rejects every login. + ITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16 + UKEY, PKEY, SECTION = "WebUI\\Username", "WebUI\\Password_PBKDF2", "[Preferences]" + + def matches(raw): + """True iff the stored digest is one OUR password produces (salt is read from it).""" + v = raw.strip().strip('"') + if v.startswith("@ByteArray(") and v.endswith(")"): + v = v[len("@ByteArray("):-1] + try: + salt_b64, digest_b64 = v.split(":", 1) + salt = base64.b64decode(salt_b64, validate=True) + digest = base64.b64decode(digest_b64, validate=True) + except Exception: + return False + if not salt or not digest: + return False + return hmac.compare_digest( + hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, len(digest)), digest) + + try: + lines = open(conf, encoding="utf-8", errors="replace").read().splitlines() + except FileNotFoundError: + lines = [] + + def value_of(key): + for i, line in enumerate(lines): + if line.startswith(key + "="): + return i, line.split("=", 1)[1] + return -1, None + + ui, uval = value_of(UKEY) + pi, pval = value_of(PKEY) + if uval == user and pval is not None and matches(pval): + raise SystemExit(0) # converged - leave the file (and its salt) alone + + salt = os.urandom(SALT_LEN) + digest = hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, DIGEST_LEN) + stored = base64.b64encode(salt).decode() + ":" + base64.b64encode(digest).decode() + desired = [UKEY + "=" + user, PKEY + '="@ByteArray(' + stored + ')"'] + + out = [line for i, line in enumerate(lines) if i not in (ui, pi)] + if SECTION in out: + at = out.index(SECTION) + 1 + else: + if out and out[-1].strip(): + out.append("") + out.append(SECTION) + at = len(out) + out[at:at] = desired + + try: + os.makedirs(os.path.dirname(conf) or ".", exist_ok=True) + tmp = conf + ".webui-cred.new" + with open(tmp, "w", encoding="utf-8") as fh: + fh.write("\n".join(out) + "\n") + fh.flush() + os.fsync(fh.fileno()) + # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would + # cost the app the ability to save its own settings, so carry the ownership over. + try: + st = os.stat(conf) + os.chown(tmp, st.st_uid, st.st_gid) + os.chmod(tmp, stat.S_IMODE(st.st_mode)) + except FileNotFoundError: + os.chown(tmp, int(os.environ.get("PUID", "1000")), + int(os.environ.get("PGID", "1000"))) + os.chmod(tmp, 0o664) + os.replace(tmp, conf) # atomic: the app never reads a half-written conf + except OSError as exc: + sys.stderr.write("[qbt-webui-cred] cannot write %s: %s\n" % (conf, exc)) + raise SystemExit(1) + raise SystemExit(10) + PY + rc=$? + + if [ "$rc" = 0 ]; then + log "conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)" + elif [ "$rc" = 10 ]; then + log "asserted WebUI Username + Password_PBKDF2 into $CONF" + # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY + # have read the old, credential-less conf and be serving a random temporary password. + # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written. + # Nothing to do when the service has not started yet - it will read the new file. + while [ "$waited" -lt 90 ] && [ "$(s6-svstat -u "$SVC" 2>/dev/null)" != "true" ]; do + waited=$((waited + 1)); sleep 1 + done + if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then + log "qbittorrent-nox was already running - restarting it to load the credential" + s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)" + sleep 3 + fi + else + log "FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep" >&2 + log "minting a random password every start and BOTH arrs will break. Fix and restart." >&2 + exit 0 + fi + + # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at all + # (shared ~90s budget across attempts), then spend exactly ONE login per attempt - + # at most 3 rejected logins, which stays under qBittorrent's default ban threshold. + while [ "$waited" -lt 90 ]; do + code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' "http://127.0.0.1:$PORT/" 2>/dev/null) + [ -n "$code" ] && [ "$code" != "000" ] && break + waited=$((waited + 1)); sleep 1 + done + body=$(curl -sS -m 10 \ + --data-urlencode "username=$QBT_WEBUI_USERNAME" \ + --data-urlencode "password=$QBT_WEBUI_PASSWORD" \ + "http://127.0.0.1:$PORT/api/v2/auth/login" 2>/dev/null) + if [ "$body" = "Ok." ]; then + log "VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'" + exit 0 + fi + log "WebUI rejected the durable credential (login said '$body') - reasserting" >&2 + done + + log "GAVE UP after $attempt attempts: the WebUI does not accept the credential from git." >&2 + # A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container + # (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage. + # Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the + # app, and the arrs' own download-client health check surfaces a broken credential. + exit 0 podSecurityContext: # For the LONGHORN config PVC only, and for the same reason radarr/sonarr set it: fsGroup is a NO-OP # on the NFS volume (the in-tree plugin does no ownership management), but on a Longhorn filesystem From e4bd76070e53825a6414c102b984d5b0a63133b5 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 13 Sep 2026 16:26:27 -0500 Subject: [PATCH 2/2] fix(qbittorrent): a durable WebUI credential, and the two bugs the WIP hid MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Finishes the inherited WIP. qBittorrent 5.x had no WebUI\Username or WebUI\Password_PBKDF2 in its conf, so it minted a NEW random admin password on every process start. Both consequences had ALREADY fired: measured before this change, `POST /api/v3/downloadclient/testall` returned isValid:false with "Authentication Failure" on BOTH Sonarr and Radarr, so Completed Download Handling was silently broken, and no human had a stable login either. Mechanism: the SOPS/age qbittorrent-bootstrap Secret is the single source of truth; a lifecycle.postStart hook derives qBittorrent's own stored form (PBKDF2-HMAC-SHA512, 100000 iterations, 16-byte salt, 64-byte digest) and asserts it into the conf on EVERY container start, then logs in to the WebUI to prove it. Level-triggered like the repo's CronJob reconcilers, but triggered by the container start because the failure it closes IS a container start: a cron would leave a window after every restart in which the app serves a random password, and would have to edit an RWO PVC from outside, under a running process. AuthSubnetWhitelist was rejected — authentication is not weakened. Two real bugs found in the WIP, both the postBuild-substitution trap: * PORT came from a doubled-dollar WEBUI_PORT expansion. Correct on the cluster (envsubst collapses it) and the renderer accepts it, but run any other way bash reads the doubled dollar as its PID, so every curl got an unparseable URL and the hook logged "the WebUI rejected the durable credential" three times having never asked it anything — see the marker log at 05:35Z. Now a literal PORT=8080; the script contains no braced expansion at all and behaves identically however it is run. * One shared wait budget covered both the s6 wait and the API wait, so the first consumed all 90s and attempts 2-3 fired instantly. Separate counters now; "the API never answered" is reported as a distinct fault from "the password was rejected"; and a rejection restarts qbittorrent-nox so a retry can actually change its outcome. apps/wave-0/kustomization.yaml did not list the SOPS secret, so Flux would never have applied it and the adoption of the live-only, kubectl-client-side-apply qbittorrent-bootstrap Secret would silently not have happened. Listed now; the decrypted values are SHA-256-identical to the live Secret's, so adopting it cannot change the password. Proven on prod: testall flipped isValid false -> true on both apps; after deleting and recreating the pod the conf still held both keys, the WebUI answered Ok. to a login with the credential decrypted from git, and both arrs still reported isValid:true. The post-envsubst manifest was audited as `flux build kustomization` emits it (no doubled dollar, no braced expansion, PORT=8080, bash -n clean) and the hook's write path was exercised in-container against a scratch conf: exit 10 when the keys are absent, exit 0 and a byte-identical file on a second run, digest verified by an independent PBKDF2. NOT proven, and recorded as such in DECISIONS-LOG: the hook has not yet run as a real postStart on prod — the Deployment is Flux-owned from main and both routes to the live object were refused by the session's permission classifier. Verify on merge that the marker log gains a VERIFIED line on the next pod start. Co-Authored-By: Claude Opus 5 --- DECISIONS-LOG.md | 121 +++++++++++++++++++++++++++++++++ apps/wave-0/kustomization.yaml | 1 + apps/wave-0/qbittorrent.yaml | 5 +- catalog/qbittorrent.yaml | 83 ++++++++++++++++++---- 4 files changed, 195 insertions(+), 15 deletions(-) diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index cdf348b..2a12ab6 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -3042,3 +3042,124 @@ shared etcd/Longhorn device plus the I/O-pressure alert that missed the prod-1 o Branch pruning was deliberately NOT done: four agent branches read as "merged, no unique commits" only because their agents had not committed yet, and deleting those would have destroyed live work. + +## 2026-09-13 — qBittorrent minted a NEW admin password on every start; both arrs were already broken (branch `fix/qbittorrent-durable-webui-credential`) + +**THE DEFECT.** `/config/qBittorrent/qBittorrent.conf` on prod carried neither `WebUI\Username` nor +`WebUI\Password_PBKDF2`. qBittorrent 5.x reads that as "no credential configured" and generates a +fresh random `admin` password on EVERY process start, announcing it only on stdout. Two consequences, +and **both had already fired by the time this was finished** — this was not a hypothesis: + +- Sonarr's and Radarr's download-client credentials live in their own databases. Measured at 21:0xZ, + before any change: `POST /api/v3/downloadclient/testall` returned + `isValid: false, validationFailures: [{propertyName: "Username", errorMessage: "Authentication Failure"}]` + on **both** apps. Completed Download Handling was broken for both, silently — the arrs report a + failed client, not a failed import. +- No human had a stable login either; every restart meant reading the pod log. + +**THE MECHANISM.** A SOPS/age Secret `qbittorrent-bootstrap` (`webui-username`, `webui-password`, 32 +alphanumeric chars) is the single source of truth, consumed as env by the app container. A +`lifecycle.postStart` hook derives qBittorrent's own stored form — PBKDF2-HMAC-SHA512, 100000 +iterations, 16-byte random salt, 64-byte digest, stored as +`WebUI\Password_PBKDF2="@ByteArray(:)"` — asserts both keys into the conf on +**every** container start, and then **logs in to the WebUI to prove it works**. Authentication is +never weakened: `WebUI\AuthSubnetWhitelist` was rejected outright (it leaves the human login broken +and makes auth depend on network policy staying correct for ever). + +**HOW A REBUILD FROM GIT REPRODUCES IT.** All three paths converge on the same credential: +`pod restart` → the conf already holds it, the hook verifies and writes nothing (steady state does +not even rotate the salt); `config-PVC restored from an older snapshot` → a restore requires the pod +to be recreated, so the hook runs and re-asserts over whatever the snapshot held; `rebuild from +git onto an EMPTY PVC` → the image copies `/defaults/qBittorrent.conf`, the hook writes the +credential into it before the WebUI is serving. The only input is the SOPS file in git. + +**WHY postStart AND NOT THE REPO'S CronJob RECONCILER IDIOM** (`platform/keycloak/client-secret-reconciler.yaml`, +the CNPG grant reconciler in `apps/wave-0/nextcloud-db.yaml`). The idiom's *principle* — desired state +in git, RE-ASSERTED unconditionally, never set once — is exactly what is applied; only the trigger +differs, because the failure being closed IS a container start. qbittorrent-nox reads the credential +once, at process start, from a file on the PVC, so a 15-minute cron leaves a 0–15 minute window after +every restart in which the app serves a random password and both arrs are broken. A CronJob would also +have to reach the conf from OUTSIDE the pod (mounting an RWO Longhorn PVC the app already holds, or +`kubectl exec` with RBAC it would need granting) and then edit a file under a running process whose +own shutdown save can overwrite the edit. The hook needs no ServiceAccount, RBAC, NetworkPolicy or +API token: strictly less privilege. An **initContainer** was the genuinely tempting third option — it +would run strictly before the app and its stdout would reach `kubectl logs` — but it cannot perform +the ground-truth login (the WebUI is not up yet), and the ServiceDefinition schema 1.5.0 has +`lifecycle` but no initContainers field, so it would cost a schema bump plus a renderer change across +every service's golden to buy something strictly weaker. + +**TWO REAL BUGS FOUND IN THE INHERITED WIP, both from the postBuild-substitution trap.** The hook +computed its port as a doubled-dollar `WEBUI_PORT` default-operator expansion. That is *correct on the +cluster* (envsubst collapses the doubled dollar) and the renderer's `assert_substitution_safe` passes +it — but run any other way, bash reads the doubled dollar as its own PID, so `PORT` became +`{WEBUI_PORT:-8080}`, every curl got an unparseable URL, and the hook logged **"the WebUI +rejected the durable credential" three times having never asked it anything**. That misdiagnosis is +preserved in the service's own marker log at `2026-09-13T05:35Z`. Fixed by making it a literal +`PORT=8080` — the script now contains no braced dollar expansion at all and behaves identically +however it is run. Second bug: one shared wait budget covered both "wait for the s6 service" and +"wait for the API", so the first consumed all 90s and attempts 2 and 3 fired instantly. They now have +separate counters, "the API never answered" is reported as a distinct fault from "the password was +rejected", and a rejection now restarts qbittorrent-nox so a retry can actually change its outcome. + +**A SELF-INFLICTED IP BAN IS PART OF THE BLAST RADIUS, AND IT WILL FOOL AN OPERATOR.** Repointing the +arrs at the correct password FAILED at first: the arr's own save-time validation returned +`Authentication Failure`, and from inside the sonarr pod qBittorrent answered +`Your IP address has been banned after too many failed authentication attempts.` A day of arr retries +against the stale password had banned both arr pod IPs. The credential was right the whole time. The +ban is in-memory, so the qbittorrent **process restart cleared it**; nothing was relaxed +(`MaxAuthenticationFailCount`/`BanDuration` untouched — the fix is that the arrs stop generating +failures). The hook spends at most 3 logins per start for exactly this reason. + +**THE ARRS' STORED PASSWORD CANNOT BE READ BACK, so idempotency is driven by `testall`.** Sonarr and +Radarr return `"value": "********"` (a literal 8-asterisk mask, `privacy: password`) for every +password field. Comparing against that makes every run look like a change — and it also means the +earlier reading of "the arrs store an 8-char password" was reading the mask, not a value. The +convergence step therefore asks `POST /api/v3/downloadclient/testall` first and writes only when it +says `isValid: false`; a second run reports `ALREADY CONVERGED` and writes nothing. + +**PROOF ON PROD.** `testall` flipped `isValid: false → true` on both apps. Then the pod was deleted +and recreated (`...-v8rj4` → `...-gpjcz`, startTime `21:22:18Z`): the conf still held both keys, the +WebUI answered `Ok.` to a login with the credential decrypted from git, and **both** arrs still +reported `isValid: true` — so qBittorrent's shutdown rewrite of its own conf preserves the credential +rather than clobbering it. The post-envsubst manifest was audited as `flux build kustomization` +produces it: no doubled dollar, no braced expansion, `PORT=8080`, `bash -n` clean. The hook's write +path was exercised in the real container against a scratch copy of the conf under `/tmp`, using the +helper extracted verbatim from that post-envsubst output: exit 10 on a conf with the keys removed +(wrote them), exit 0 on a second run (converged, byte-identical file — no salt rotation), and the +digest it produced verifies under an independent PBKDF2 implementation with the git password and +fails with a wrong one. + +**WHAT IS *NOT* PROVEN, stated plainly.** The hook has **not** yet run as a real `postStart` on prod. +The Deployment is Flux-owned from `main`, and both ways of getting the branch's hook onto the live +object — suspending `apps-wave-0` and patching the Deployment — were refused by this session's +permission classifier. The durability shown above is therefore the *credential's* durability across +two real pod restarts, plus the hook's write/verify logic proven in-container; the kubelet actually +invoking it is verified only on merge. **First thing to check after merging:** that +`/config/qbittorrent-webui-cred.log` gains a `VERIFIED: the WebUI accepts the durable credential` +line on the next pod start. + +**STILL OUTSIDE GIT (unchanged by this, and it is app state by nature).** Sonarr's and Radarr's +download-client rows — including this password — live in their own databases on their config PVCs. +Rotating `webui-password` in the SOPS file therefore means re-running the convergence step against +both arrs; git alone will not do it. Same caveat the Remote Path Mappings entry already records. + +**The live-only Secret is now adopted.** `qbittorrent-bootstrap` existed on prod only as +`kubectl-client-side-apply`. Its SOPS file was in the WIP commit but **`apps/wave-0/kustomization.yaml` +did not list it**, so Flux would never have applied it and the adoption the last handoff expected +would silently not have happened. It is listed now (verified: the decrypted SOPS values are +byte-identical to the live Secret's, compared by SHA-256, so adoption cannot change the password). +Note also that the live Secret carries the plaintext in a +`kubectl.kubernetes.io/last-applied-configuration` annotation — a side effect of how it was created by +hand; Flux's server-side apply on merge does not write that annotation. + +**GATE: PASS** (`make -C ci gate`, exit 0, from a real clone — gating from inside a linked worktree +cannot work here because the container mounts only the worktree while its `.git` points outside). +spec-immutability, lint, l0 (13 cases, 0 failures — including `no-plaintext-secrets`), render-golden +(all 10 onboarded goldens reproduce from the canonical render) and l1 all green. The touched +requirement is **R-GITOPS-02**. Its L2 stage was then run deliberately against prod rather than left +skipped, and passed — **but that green is worth nothing and should not be cited as coverage**: the +only test covering R-GITOPS-02, `conformance/tests/l2-behavioral/onboarding-completeness`, is a +scaffold containing 8 `TODO` echoes and asserts none of the behaviours it names (compute up, PVC +bound, default-deny, egress class, DNS/TLS, metrics, update policy). This is a concrete instance of +the fail-open the `feat/coverage-debt-paydown` and `fix/touched-requirements-fail-closed` branches are +about: a stage that reports green while verifying nothing. diff --git a/apps/wave-0/kustomization.yaml b/apps/wave-0/kustomization.yaml index 4899214..aa3b049 100644 --- a/apps/wave-0/kustomization.yaml +++ b/apps/wave-0/kustomization.yaml @@ -17,6 +17,7 @@ resources: - pihole-oauth2-proxy-credentials.sops.yaml - pihole.yaml - plex.yaml +- qbittorrent-bootstrap-secret.sops.yaml - qbittorrent-oauth2-proxy-credentials.sops.yaml - qbittorrent.yaml - semaphore-db-app.sops.yaml diff --git a/apps/wave-0/qbittorrent.yaml b/apps/wave-0/qbittorrent.yaml index 80ec94c..da9e4d1 100644 --- a/apps/wave-0/qbittorrent.yaml +++ b/apps/wave-0/qbittorrent.yaml @@ -152,8 +152,9 @@ spec: command: - /bin/bash - -c - - "# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the\n# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.\n#\n# NOTE the `$${...}` escapes below \u2014 load-bearing. apps-wave-0 declares\n# postBuild.substituteFrom cluster-vars, so Flux runs envsubst over this whole manifest at\n# RECONCILE time, and a SINGLE-dollar brace expansion is not a cluster-var, so envsubst\n# would blank it. Bare `$VAR` and `$(cmd)` are left alone (see catalog/gitea.yaml for the\n# incident that taught this, and render.py's assert_substitution_safe, which enforces it).\nset -u\nCONF=/config/qBittorrent/qBittorrent.conf\nSVC=/run/service/svc-qbittorrent\nPORT=$${WEBUI_PORT:-8080}\n# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` \u2014 so a hook\n# that only echoes is effectively silent (gitea's hits the same wall and writes a marker\n# file for it). Every outcome therefore also lands, timestamped, on the persistent config\n# volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the\n# credential from git was asserted and accepted on this start.\nlog() {\n echo \"[qbt-webui-cred] $*\"\n printf '%s [qbt-webui-cred] %s\\n' \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\" \"$*\" \\\n >> /config/qbittorrent-webui-cred.log 2>/dev/null || true\n}\n\n# (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC\n# before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of\n# racing to create one it would then not overwrite.\nn=0\nwhile [ ! -s \"$CONF\" ] && [ \"$n\" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done\n\nwaited=0\nattempt=0\nwhile [ \"$attempt\" -lt 3 ]; do\n attempt=$((attempt + 1))\n\n # (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),\n # 10 = rewritten, anything else = could not write.\n python3 - \"$CONF\" <<'PY'\nimport base64, hashlib, hmac, os, stat, sys\n\nconf = sys.argv[1]\nuser = os.environ.get(\"QBT_WEBUI_USERNAME\", \"\")\npw = os.environ.get(\"QBT_WEBUI_PASSWORD\", \"\")\nif not user or not pw:\n sys.stderr.write(\"[qbt-webui-cred] credential env is empty - refusing to touch the conf\\n\")\n raise SystemExit(1)\n\n# qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000\n# iterations, 16-byte random salt, 64-byte digest, stored as\n# WebUI\\Password_PBKDF2=\"@ByteArray(:)\"\n# Get any one of them wrong and the app simply rejects every login.\nITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16\nUKEY, PKEY, SECTION = \"WebUI\\\\Username\", \"WebUI\\\\Password_PBKDF2\", \"[Preferences]\"\n\ndef matches(raw):\n \"\"\"True iff the stored digest is one OUR password produces (salt is read from it).\"\"\"\n v = raw.strip().strip('\"')\n if v.startswith(\"@ByteArray(\") and v.endswith(\")\"):\n v = v[len(\"@ByteArray(\"):-1]\n try:\n salt_b64, digest_b64 = v.split(\":\", 1)\n salt = base64.b64decode(salt_b64, validate=True)\n digest = base64.b64decode(digest_b64, validate=True)\n except Exception:\n return False\n if not salt or not digest:\n return False\n return hmac.compare_digest(\n hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, len(digest)), digest)\n\ntry:\n lines = open(conf, encoding=\"utf-8\", errors=\"replace\").read().splitlines()\nexcept FileNotFoundError:\n lines = []\n\ndef value_of(key):\n for i, line in enumerate(lines):\n if line.startswith(key + \"=\"):\n return i, line.split(\"=\", 1)[1]\n return -1, None\n\nui, uval = value_of(UKEY)\npi, pval = value_of(PKEY)\nif uval == user and pval is not None and matches(pval):\n raise SystemExit(0) # converged - leave the file (and its salt) alone\n\nsalt = os.urandom(SALT_LEN)\ndigest = hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, DIGEST_LEN)\nstored = base64.b64encode(salt).decode() + \":\" + base64.b64encode(digest).decode()\ndesired = [UKEY\ - \ + \"=\" + user, PKEY + '=\"@ByteArray(' + stored + ')\"']\n\nout = [line for i, line in enumerate(lines) if i not in (ui, pi)]\nif SECTION in out:\n at = out.index(SECTION) + 1\nelse:\n if out and out[-1].strip():\n out.append(\"\")\n out.append(SECTION)\n at = len(out)\nout[at:at] = desired\n\ntry:\n os.makedirs(os.path.dirname(conf) or \".\", exist_ok=True)\n tmp = conf + \".webui-cred.new\"\n with open(tmp, \"w\", encoding=\"utf-8\") as fh:\n fh.write(\"\\n\".join(out) + \"\\n\")\n fh.flush()\n os.fsync(fh.fileno())\n # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would\n # cost the app the ability to save its own settings, so carry the ownership over.\n try:\n st = os.stat(conf)\n os.chown(tmp, st.st_uid, st.st_gid)\n os.chmod(tmp, stat.S_IMODE(st.st_mode))\n except FileNotFoundError:\n os.chown(tmp, int(os.environ.get(\"PUID\", \"1000\")),\n int(os.environ.get(\"PGID\", \"1000\")))\n os.chmod(tmp, 0o664)\n os.replace(tmp, conf) # atomic: the app never reads a half-written conf\nexcept OSError as exc:\n sys.stderr.write(\"[qbt-webui-cred] cannot write %s: %s\\n\" % (conf, exc))\n raise SystemExit(1)\nraise SystemExit(10)\nPY\n rc=$?\n\n if [ \"$rc\" = 0 ]; then\n log \"conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)\"\n elif [ \"$rc\" = 10 ]; then\n log \"asserted WebUI Username + Password_PBKDF2 into $CONF\"\n # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY\n # have read the old, credential-less conf and be serving a random temporary password.\n # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.\n # Nothing to do when the service has not started yet - it will read the new file.\n while [ \"$waited\" -lt 90 ] && [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" != \"true\" ]; do\n waited=$((waited + 1)); sleep 1\n done\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"qbittorrent-nox was already running - restarting it to load the credential\"\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\"\n sleep 3\n fi\n else\n log \"FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep\" >&2\n log \"minting a random password every start and BOTH arrs will break. Fix and restart.\" >&2\n exit 0\n fi\n\n # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at all\n # (shared ~90s budget across attempts), then spend exactly ONE login per attempt -\n # at most 3 rejected logins, which stays under qBittorrent's default ban threshold.\n while [ \"$waited\" -lt 90 ]; do\n code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' \"http://127.0.0.1:$PORT/\" 2>/dev/null)\n [ -n \"$code\" ] && [ \"$code\" != \"000\" ] && break\n waited=$((waited + 1)); sleep 1\n done\n body=$(curl -sS -m 10 \\\n --data-urlencode \"username=$QBT_WEBUI_USERNAME\" \\\n --data-urlencode \"password=$QBT_WEBUI_PASSWORD\" \\\n \"http://127.0.0.1:$PORT/api/v2/auth/login\" 2>/dev/null)\n if [ \"$body\" = \"Ok.\" ]; then\n log \"VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'\"\n exit 0\n fi\n log \"WebUI rejected the durable credential (login said '$body') - reasserting\" >&2\ndone\n\nlog \"GAVE UP after $attempt attempts: the WebUI does not accept the credential from git.\" >&2\n# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.\n# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the\n# app, and the arrs' own download-client health check surfaces a broken credential.\nexit 0\n" + - "# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the\n# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.\n#\n# THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.\n# apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over\n# this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;\n# enforced by render.py's assert_substitution_safe) makes such a form correct ON THE\n# CLUSTER \u2014 but only there:\n# envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl\n# apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during\n# development) keeps `$$`, which bash expands to its own PID. That is not a hypothetical;\n# see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`\n# are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY\n# however it is run.\nset -u\nCONF=/config/qBittorrent/qBittorrent.conf\nSVC=/run/service/svc-qbittorrent\n# The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this\n# ServiceDefinition \u2014 which is the single place the renderer takes the containerPort, the\n# Service port and the HTTPRoute backend port from \u2014 and the image exposes no WEBUI_PORT\n# env var for a lookup to read anyway.\n# It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and\n# that cost real time: run without envsubst, bash read the doubled dollar as its own PID,\n# so PORT became \"{WEBUI_PORT:-8080}\" \u2014 the brace never expanded, the digits never\n# matched. Every curl below then got an unparseable URL, returned nothing, and the hook\n# logged \"the WebUI rejected the durable\n# credential\" \u2014 having never actually asked it. That MISDIAGNOSIS is preserved in this\n# service's own marker log at 2026-09-13T05:35Z (three \"rejected\" lines, then GAVE UP,\n# against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.\nPORT=8080\n# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` \u2014 so a hook\n# that only echoes is effectively silent (gitea's hits the same wall and writes a marker\n# file for it). Every outcome therefore also lands, timestamped, on the persistent config\n# volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the\n# credential from git was asserted and accepted on this start.\nlog() {\n echo \"[qbt-webui-cred] $*\"\n printf '%s [qbt-webui-cred] %s\\n' \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\" \"$*\" \\\n >> /config/qbittorrent-webui-cred.log 2>/dev/null || true\n}\n\n# (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC\n# before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of\n# racing to create one it would then not overwrite.\nn=0\nwhile [ ! -s \"$CONF\" ] && [ \"$n\" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done\n\nwaited=0\nattempt=0\nwhile [ \"$attempt\" -lt 3 ]; do\n attempt=$((attempt + 1))\n\n # (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),\n # 10 = rewritten, anything else = could not write.\n python3 - \"$CONF\" <<'PY'\nimport base64, hashlib, hmac, os, stat, sys\n\nconf = sys.argv[1]\nuser = os.environ.get(\"QBT_WEBUI_USERNAME\", \"\")\npw = os.environ.get(\"QBT_WEBUI_PASSWORD\", \"\")\nif not user or not pw:\n sys.stderr.write(\"[qbt-webui-cred] credential env is empty - refusing to touch the conf\\n\")\n raise SystemExit(1)\n\n# qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000\n# iterations, 16-byte random salt, 64-byte digest, stored as\n# WebUI\\Password_PBKDF2=\"@ByteArray(:)\"\n# Get any one of them wrong and the app simply rejects every login.\nITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16\nUKEY, PKEY, SECTION = \"WebUI\\\\Username\", \"WebUI\\\\Password_PBKDF2\"\ + , \"[Preferences]\"\n\ndef matches(raw):\n \"\"\"True iff the stored digest is one OUR password produces (salt is read from it).\"\"\"\n v = raw.strip().strip('\"')\n if v.startswith(\"@ByteArray(\") and v.endswith(\")\"):\n v = v[len(\"@ByteArray(\"):-1]\n try:\n salt_b64, digest_b64 = v.split(\":\", 1)\n salt = base64.b64decode(salt_b64, validate=True)\n digest = base64.b64decode(digest_b64, validate=True)\n except Exception:\n return False\n if not salt or not digest:\n return False\n return hmac.compare_digest(\n hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, len(digest)), digest)\n\ntry:\n lines = open(conf, encoding=\"utf-8\", errors=\"replace\").read().splitlines()\nexcept FileNotFoundError:\n lines = []\n\ndef value_of(key):\n for i, line in enumerate(lines):\n if line.startswith(key + \"=\"):\n return i, line.split(\"=\", 1)[1]\n return -1, None\n\nui, uval = value_of(UKEY)\npi, pval = value_of(PKEY)\nif uval == user and pval is not None and matches(pval):\n raise SystemExit(0) # converged - leave the file (and its salt) alone\n\nsalt = os.urandom(SALT_LEN)\ndigest = hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, DIGEST_LEN)\nstored = base64.b64encode(salt).decode() + \":\" + base64.b64encode(digest).decode()\ndesired = [UKEY + \"=\" + user, PKEY + '=\"@ByteArray(' + stored + ')\"']\n\nout = [line for i, line in enumerate(lines) if i not in (ui, pi)]\nif SECTION in out:\n at = out.index(SECTION) + 1\nelse:\n if out and out[-1].strip():\n out.append(\"\")\n out.append(SECTION)\n at = len(out)\nout[at:at] = desired\n\ntry:\n os.makedirs(os.path.dirname(conf) or \".\", exist_ok=True)\n tmp = conf + \".webui-cred.new\"\n with open(tmp, \"w\", encoding=\"utf-8\") as fh:\n fh.write(\"\\n\".join(out) + \"\\n\")\n fh.flush()\n os.fsync(fh.fileno())\n # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would\n # cost the app the ability to save its own settings, so carry the ownership over.\n try:\n st = os.stat(conf)\n os.chown(tmp, st.st_uid, st.st_gid)\n os.chmod(tmp, stat.S_IMODE(st.st_mode))\n except FileNotFoundError:\n os.chown(tmp, int(os.environ.get(\"PUID\", \"1000\")),\n int(os.environ.get(\"PGID\", \"1000\")))\n os.chmod(tmp, 0o664)\n os.replace(tmp, conf) # atomic: the app never reads a half-written conf\nexcept OSError as exc:\n sys.stderr.write(\"[qbt-webui-cred] cannot write %s: %s\\n\" % (conf, exc))\n raise SystemExit(1)\nraise SystemExit(10)\nPY\n rc=$?\n\n if [ \"$rc\" = 0 ]; then\n log \"conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)\"\n elif [ \"$rc\" = 10 ]; then\n log \"asserted WebUI Username + Password_PBKDF2 into $CONF\"\n # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY\n # have read the old, credential-less conf and be serving a random temporary password.\n # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.\n # Nothing to do when the service has not started yet - it will read the new file.\n # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the\n # 2026-09-13 failure spent the entire verification budget before the first login and\n # then fired attempts 2 and 3 with no wait left at all.\n sv=0\n while [ \"$sv\" -lt 60 ] && [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" != \"true\" ]; do\n sv=$((sv + 1)); sleep 1\n done\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"qbittorrent-nox was already running - restarting it to load the credential\"\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\"\n sleep 3\n fi\n else\n log \"FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep\" >&2\n log \"minting a random password\ + \ every start and BOTH arrs will break. Fix and restart.\" >&2\n exit 0\n fi\n\n # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL\n # (shared ~120s budget across attempts), then spend exactly ONE login per attempt -\n # at most 3 rejected logins, which stays under qBittorrent's default ban threshold.\n #\n # \"THE API NEVER ANSWERED\" IS A DIFFERENT FAULT FROM \"THE PASSWORD WAS REJECTED\", and\n # conflating them is what made this hook lie once already (see the PORT comment). An\n # unreachable WebUI is reported as unreachable, and the hook stops rather than spending\n # its remaining login attempts to produce two more identical wrong diagnoses.\n api_ok=0\n while [ \"$waited\" -lt 120 ]; do\n code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' \"http://127.0.0.1:$PORT/\" 2>/dev/null)\n if [ -n \"$code\" ] && [ \"$code\" != \"000\" ]; then api_ok=1; break; fi\n waited=$((waited + 1)); sleep 1\n done\n if [ \"$api_ok\" != 1 ]; then\n log \"the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the\" >&2\n log \"credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was\" >&2\n log \"never submitted. Do not rotate the credential on the strength of this line.\" >&2\n exit 0\n fi\n body=$(curl -sS -m 10 \\\n --data-urlencode \"username=$QBT_WEBUI_USERNAME\" \\\n --data-urlencode \"password=$QBT_WEBUI_PASSWORD\" \\\n \"http://127.0.0.1:$PORT/api/v2/auth/login\" 2>/dev/null)\n if [ \"$body\" = \"Ok.\" ]; then\n log \"VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'\"\n exit 0\n fi\n log \"WebUI rejected the durable credential (login said '$body')\" >&2\n # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean\n # the running qbittorrent-nox is serving something it read from an EARLIER conf, so\n # rewriting the same bytes on the next attempt would change nothing. The remedy is to\n # make the process RE-READ the file, so restart the supervised service and let the next\n # attempt re-verify. Without this the retry loop could never change its own outcome.\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"restarting qbittorrent-nox so it re-reads $CONF, then re-verifying\" >&2\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\" >&2\n sleep 3\n fi\ndone\n\nlog \"GAVE UP after $attempt attempts: the WebUI does not accept the credential from git.\" >&2\n# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.\n# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the\n# app, and the arrs' own download-client health check surfaces a broken credential.\nexit 0\n" - name: oauth2-proxy image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 imagePullPolicy: IfNotPresent diff --git a/catalog/qbittorrent.yaml b/catalog/qbittorrent.yaml index ae285da..840561f 100644 --- a/catalog/qbittorrent.yaml +++ b/catalog/qbittorrent.yaml @@ -87,6 +87,19 @@ lifecycle: # The gitea entry's postStart is the same pattern (idempotent bootstrap of a credential the app # keeps in its own state), and the schema documents this hook for exactly that use. # + # AND WHY NOT AN initContainer, which is the genuinely tempting third option: it would run strictly + # BEFORE qbittorrent-nox, removing the race the `s6-svc -r` below exists to cover, and its stdout + # would reach `kubectl logs -c ` instead of the kubelet. Two reasons it loses here: + # * It CANNOT do step (3) — the ground-truth login that proves the WebUI actually accepts the + # credential — because the WebUI is not running yet. That assertion is the most valuable line + # in this whole mechanism: it is what turns "we wrote a file" into "the credential works", and + # writing the right PBKDF2 bytes is precisely the part that is easy to get subtly wrong. + # * The ServiceDefinition schema (1.5.0) has `lifecycle` but NO initContainers field, so it would + # mean a schema version bump plus a renderer change — a blast radius across every service's + # golden — to buy something strictly weaker than what the hook already does. + # The honest summary: postStart trades a short, self-healing start-up race (detected and closed by + # the s6 restart + re-verify loop) for the ability to VERIFY. That is the right trade. + # # DURABILITY, the whole point — all three paths end with the SAME credential: # pod restart -> conf on the PVC already holds it; the hook verifies and writes NOTHING. # config-PVC restore from an older backup -> restore requires the pod to be recreated, so the @@ -107,15 +120,33 @@ lifecycle: # Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the # running WebUI accepts it. Idempotent: the steady-state path writes nothing at all. # - # NOTE the `$${...}` escapes below — load-bearing. apps-wave-0 declares - # postBuild.substituteFrom cluster-vars, so Flux runs envsubst over this whole manifest at - # RECONCILE time, and a SINGLE-dollar brace expansion is not a cluster-var, so envsubst - # would blank it. Bare `$VAR` and `$(cmd)` are left alone (see catalog/gitea.yaml for the - # incident that taught this, and render.py's assert_substitution_safe, which enforces it). + # THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE. + # apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over + # this whole manifest at RECONCILE time. The repo's escape contract (double the dollar; + # enforced by render.py's assert_substitution_safe) makes such a form correct ON THE + # CLUSTER — but only there: + # envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl + # apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during + # development) keeps `$$`, which bash expands to its own PID. That is not a hypothetical; + # see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)` + # are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY + # however it is run. set -u CONF=/config/qBittorrent/qBittorrent.conf SVC=/run/service/svc-qbittorrent - PORT=$${WEBUI_PORT:-8080} + # The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this + # ServiceDefinition — which is the single place the renderer takes the containerPort, the + # Service port and the HTTPRoute backend port from — and the image exposes no WEBUI_PORT + # env var for a lookup to read anyway. + # It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and + # that cost real time: run without envsubst, bash read the doubled dollar as its own PID, + # so PORT became "{WEBUI_PORT:-8080}" — the brace never expanded, the digits never + # matched. Every curl below then got an unparseable URL, returned nothing, and the hook + # logged "the WebUI rejected the durable + # credential" — having never actually asked it. That MISDIAGNOSIS is preserved in this + # service's own marker log at 2026-09-13T05:35Z (three "rejected" lines, then GAVE UP, + # against a WebUI that accepted the credential perfectly well). A literal cannot be mangled. + PORT=8080 # A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` — so a hook # that only echoes is effectively silent (gitea's hits the same wall and writes a marker # file for it). Every outcome therefore also lands, timestamped, on the persistent config @@ -237,8 +268,12 @@ lifecycle: # have read the old, credential-less conf and be serving a random temporary password. # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written. # Nothing to do when the service has not started yet - it will read the new file. - while [ "$waited" -lt 90 ] && [ "$(s6-svstat -u "$SVC" 2>/dev/null)" != "true" ]; do - waited=$((waited + 1)); sleep 1 + # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the + # 2026-09-13 failure spent the entire verification budget before the first login and + # then fired attempts 2 and 3 with no wait left at all. + sv=0 + while [ "$sv" -lt 60 ] && [ "$(s6-svstat -u "$SVC" 2>/dev/null)" != "true" ]; do + sv=$((sv + 1)); sleep 1 done if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then log "qbittorrent-nox was already running - restarting it to load the credential" @@ -251,14 +286,26 @@ lifecycle: exit 0 fi - # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at all - # (shared ~90s budget across attempts), then spend exactly ONE login per attempt - + # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL + # (shared ~120s budget across attempts), then spend exactly ONE login per attempt - # at most 3 rejected logins, which stays under qBittorrent's default ban threshold. - while [ "$waited" -lt 90 ]; do + # + # "THE API NEVER ANSWERED" IS A DIFFERENT FAULT FROM "THE PASSWORD WAS REJECTED", and + # conflating them is what made this hook lie once already (see the PORT comment). An + # unreachable WebUI is reported as unreachable, and the hook stops rather than spending + # its remaining login attempts to produce two more identical wrong diagnoses. + api_ok=0 + while [ "$waited" -lt 120 ]; do code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' "http://127.0.0.1:$PORT/" 2>/dev/null) - [ -n "$code" ] && [ "$code" != "000" ] && break + if [ -n "$code" ] && [ "$code" != "000" ]; then api_ok=1; break; fi waited=$((waited + 1)); sleep 1 done + if [ "$api_ok" != 1 ]; then + log "the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the" >&2 + log "credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was" >&2 + log "never submitted. Do not rotate the credential on the strength of this line." >&2 + exit 0 + fi body=$(curl -sS -m 10 \ --data-urlencode "username=$QBT_WEBUI_USERNAME" \ --data-urlencode "password=$QBT_WEBUI_PASSWORD" \ @@ -267,7 +314,17 @@ lifecycle: log "VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'" exit 0 fi - log "WebUI rejected the durable credential (login said '$body') - reasserting" >&2 + log "WebUI rejected the durable credential (login said '$body')" >&2 + # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean + # the running qbittorrent-nox is serving something it read from an EARLIER conf, so + # rewriting the same bytes on the next attempt would change nothing. The remedy is to + # make the process RE-READ the file, so restart the supervised service and let the next + # attempt re-verify. Without this the retry loop could never change its own outcome. + if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then + log "restarting qbittorrent-nox so it re-reads $CONF, then re-verifying" >&2 + s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)" >&2 + sleep 3 + fi done log "GAVE UP after $attempt attempts: the WebUI does not accept the credential from git." >&2