jakman85andClaude Opus 5
e4bd76070e
Finishes the inherited WIP. qBittorrent 5.x had no WebUI\Username or
WebUI\Password_PBKDF2 in its conf, so it minted a NEW random admin password on
every process start. Both consequences had ALREADY fired: measured before this
change, `POST /api/v3/downloadclient/testall` returned isValid:false with
"Authentication Failure" on BOTH Sonarr and Radarr, so Completed Download
Handling was silently broken, and no human had a stable login either.
Mechanism: the SOPS/age qbittorrent-bootstrap Secret is the single source of
truth; a lifecycle.postStart hook derives qBittorrent's own stored form
(PBKDF2-HMAC-SHA512, 100000 iterations, 16-byte salt, 64-byte digest) and
asserts it into the conf on EVERY container start, then logs in to the WebUI to
prove it. Level-triggered like the repo's CronJob reconcilers, but triggered by
the container start because the failure it closes IS a container start: a cron
would leave a window after every restart in which the app serves a random
password, and would have to edit an RWO PVC from outside, under a running
process. AuthSubnetWhitelist was rejected — authentication is not weakened.
Two real bugs found in the WIP, both the postBuild-substitution trap:
* PORT came from a doubled-dollar WEBUI_PORT expansion. Correct on the cluster
(envsubst collapses it) and the renderer accepts it, but run any other way
bash reads the doubled dollar as its PID, so every curl got an unparseable
URL and the hook logged "the WebUI rejected the durable credential" three
times having never asked it anything — see the marker log at 05:35Z. Now a
literal PORT=8080; the script contains no braced expansion at all and
behaves identically however it is run.
* One shared wait budget covered both the s6 wait and the API wait, so the
first consumed all 90s and attempts 2-3 fired instantly. Separate counters
now; "the API never answered" is reported as a distinct fault from "the
password was rejected"; and a rejection restarts qbittorrent-nox so a retry
can actually change its outcome.
apps/wave-0/kustomization.yaml did not list the SOPS secret, so Flux would never
have applied it and the adoption of the live-only, kubectl-client-side-apply
qbittorrent-bootstrap Secret would silently not have happened. Listed now; the
decrypted values are SHA-256-identical to the live Secret's, so adopting it
cannot change the password.
Proven on prod: testall flipped isValid false -> true on both apps; after
deleting and recreating the pod the conf still held both keys, the WebUI answered
Ok. to a login with the credential decrypted from git, and both arrs still
reported isValid:true. The post-envsubst manifest was audited as `flux build
kustomization` emits it (no doubled dollar, no braced expansion, PORT=8080,
bash -n clean) and the hook's write path was exercised in-container against a
scratch conf: exit 10 when the keys are absent, exit 0 and a byte-identical file
on a second run, digest verified by an independent PBKDF2.
NOT proven, and recorded as such in DECISIONS-LOG: the hook has not yet run as a
real postStart on prod — the Deployment is Flux-owned from main and both routes
to the live object were refused by the session's permission classifier. Verify
on merge that the marker log gains a VERIFIED line on the next pod start.
Co-Authored-By: Claude Opus 5 <[email protected]>
462 lines
31 KiB
YAML
462 lines
31 KiB
YAML
# ServiceDefinition: qbittorrent
|
|
# Source of truth: spec docs/13-service-catalog.md row 13; docs/03-networking-isolation-and-egress.md;
|
|
# docs/15-identity-and-access.md; decisions/adr-0015 (VPN egress steering).
|
|
# internal download client | egress VPN + kill-switch | config on Longhorn | auth forward-auth
|
|
# (bare web UI, no native OIDC).
|
|
#
|
|
# BULK DOWNLOADS TIER — DELIVERED 2026-09-12 (the deferral below is CLOSED; kept for the record).
|
|
# This entry used to ship config-only, with a comment promising to attach the downloads volume "once
|
|
# OQ-2 is resolved". OQ-2 was resolved for STATIC pre-existing datasets (schema 1.5.0 / renderer
|
|
# `class: bulk` + `export` -> a static, pre-bound NFS PV), and plex/sonarr/radarr/nextcloud all took
|
|
# their media mounts — but qBittorrent's was never added, so the DOWNLOAD CLIENT was the one member
|
|
# of the chain with no media volume at all. Measured on prod 2026-09-12:
|
|
# Session\DefaultSavePath=/downloads/ Session\TempPath=/downloads/incomplete/
|
|
# Downloads\SavePath=/downloads/ `ls /downloads` -> No such file or directory
|
|
# i.e. the app was configured to write a path that DID NOT EXIST, in a container with no volume there.
|
|
# Anything it did manage to create would have been invisible to sonarr/radarr, lost on every pod
|
|
# recreation, and filling the NODE's ephemeral disk in the meantime. The `downloads` entry under
|
|
# storage: below closes that. See docs/BULK-NFS-STORAGE.md and the bulk-tier L0 conformance test.
|
|
#
|
|
# VPN-class (docs/13 row 13; docs/03 "VPN-class set"): ALL of this pod's public-internet egress
|
|
# (peer/tracker/DHT traffic) MUST leave via the node PIA tunnel, never the WAN. `egress: vpn` renders
|
|
# the homelab.keithjackson.dev/egress-class=vpn pod label the shared CiliumEgressGatewayPolicy
|
|
# (platform/vpn-egress/egress-steering.yaml) selects on to steer + kill-switch the traffic OUTSIDE the
|
|
# workload (R-NET-05/06/07). No independent egress path; fail-closed if tun0 is down.
|
|
#
|
|
# Inbound reachability (<- sonarr, <- radarr) is an EAST-WEST allow rendered on the arr side; being
|
|
# VPN-class governs egress, not intra-cluster reachability. This service initiates nothing east-west, so
|
|
# dependencies is empty (reaches only DNS + VPN internet).
|
|
# Validates against schemas/servicedefinition.schema.json.
|
|
name: qbittorrent
|
|
kind: container
|
|
source:
|
|
image: lscr.io/linuxserver/qbittorrent:5.1.2 # pinned stable tag (R-UPD-01); update.range governs advancement
|
|
# LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
|
|
# LOAD-BEARING FOR THE BULK MOUNT, not cosmetic — exactly as on sonarr/radarr. Measured on prod
|
|
# 2026-09-12 BEFORE this change: `id` inside the container reported the qbittorrent-nox process as
|
|
# uid=911 gid=911, and /mnt/Storage/plex/downloads is 0755 owned 1000:1000. 0755 grants write to the
|
|
# OWNER ONLY, so uid 911 would have mounted the export cleanly and then failed every write with
|
|
# EACCES — the "mounts fine, cannot write" failure the sonarr entry warns about at length.
|
|
env:
|
|
- name: PUID
|
|
value: "1000" # owner uid of /mnt/Storage/plex/downloads (and of every dir under the export)
|
|
- name: PGID
|
|
value: "3000" # the export's dominant group, matching sonarr/radarr: files this writes stay
|
|
# readable to the EXISTING non-k3s consumers of the same dataset, and land
|
|
# group-owned the same way the arrs' own writes do
|
|
# --- THE DURABLE WebUI CREDENTIAL (consumed by the postStart hook below) -------------------
|
|
# Measured on prod 2026-09-13: /config/qBittorrent/qBittorrent.conf held NEITHER `WebUI\Username`
|
|
# NOR `WebUI\Password_PBKDF2`. qBittorrent 5.x reads that as "no credential configured" and mints a
|
|
# NEW random `admin` password on EVERY process start, announcing it ONLY on stdout ("A temporary
|
|
# password is provided for this session: ..."). Two real consequences, not hypotheticals:
|
|
# * Sonarr and Radarr store a download-client username/password in their OWN databases. Both
|
|
# tested valid against the then-current temporary password; both would have started failing at
|
|
# the NEXT qbittorrent pod restart, silently breaking Completed Download Handling (the arrs
|
|
# report a failed client, not a failed import).
|
|
# * The human operator had no stable login: every restart meant reading the pod log.
|
|
# Provided as apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml (SOPS/age, decrypted by Flux) —
|
|
# same shape and naming as gitea-bootstrap, so `render.py --emit-waves` places it in this wave.
|
|
- name: QBT_WEBUI_USERNAME
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: qbittorrent-bootstrap
|
|
key: webui-username
|
|
- name: QBT_WEBUI_PASSWORD
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: qbittorrent-bootstrap
|
|
key: webui-password
|
|
lifecycle:
|
|
# LEVEL-TRIGGERED CREDENTIAL ASSERTION, run on EVERY container start.
|
|
#
|
|
# WHY A postStart HOOK AND NOT A CronJob RECONCILER: the repo's idiom for "state Flux cannot reach
|
|
# directly" is the level-triggered CronJob (platform/keycloak/client-secret-reconciler.yaml,
|
|
# apps/wave-0/nextcloud-db.yaml). That idiom's PRINCIPLE — desired state lives in git and is
|
|
# RE-ASSERTED unconditionally, never set once — is exactly what is applied here, but the trigger is
|
|
# the container start rather than a clock, because the failure this closes IS a container start:
|
|
# * The credential is read by qbittorrent-nox ONCE, at process start, from a file on the config
|
|
# PVC. A 15-minute cron would therefore leave a 0-15 minute window after every restart, PVC
|
|
# restore or rebuild in which the app serves a random password and both arrs are broken. The
|
|
# hook has no such window: it runs before/with the very start that would otherwise regenerate.
|
|
# * A CronJob would also have to reach the conf from OUTSIDE the pod — mounting an RWO Longhorn
|
|
# PVC that the app already holds, or `kubectl exec`-ing with RBAC it would need granting — and
|
|
# then edit a file UNDER the running process, whose own save-on-exit can overwrite the edit.
|
|
# In-container, before the service starts, none of that exists.
|
|
# * It needs no ServiceAccount, no RBAC, no NetworkPolicy and no API token: strictly less
|
|
# privilege than the CronJob alternative (R-SEC/least-privilege).
|
|
# The gitea entry's postStart is the same pattern (idempotent bootstrap of a credential the app
|
|
# keeps in its own state), and the schema documents this hook for exactly that use.
|
|
#
|
|
# AND WHY NOT AN initContainer, which is the genuinely tempting third option: it would run strictly
|
|
# BEFORE qbittorrent-nox, removing the race the `s6-svc -r` below exists to cover, and its stdout
|
|
# would reach `kubectl logs -c <init>` instead of the kubelet. Two reasons it loses here:
|
|
# * It CANNOT do step (3) — the ground-truth login that proves the WebUI actually accepts the
|
|
# credential — because the WebUI is not running yet. That assertion is the most valuable line
|
|
# in this whole mechanism: it is what turns "we wrote a file" into "the credential works", and
|
|
# writing the right PBKDF2 bytes is precisely the part that is easy to get subtly wrong.
|
|
# * The ServiceDefinition schema (1.5.0) has `lifecycle` but NO initContainers field, so it would
|
|
# mean a schema version bump plus a renderer change — a blast radius across every service's
|
|
# golden — to buy something strictly weaker than what the hook already does.
|
|
# The honest summary: postStart trades a short, self-healing start-up race (detected and closed by
|
|
# the s6 restart + re-verify loop) for the ability to VERIFY. That is the right trade.
|
|
#
|
|
# DURABILITY, the whole point — all three paths end with the SAME credential:
|
|
# pod restart -> conf on the PVC already holds it; the hook verifies and writes NOTHING.
|
|
# config-PVC restore from an older backup -> restore requires the pod to be recreated, so the
|
|
# hook runs and re-asserts over whatever the snapshot contained.
|
|
# rebuild from git -> empty PVC; the image copies /defaults/qBittorrent.conf, the hook writes the
|
|
# credential into it before the service is serving. Nothing outside git.
|
|
#
|
|
# WHAT IT DOES NOT DO: it never weakens authentication. `WebUI\AuthSubnetWhitelist` (bypassing auth
|
|
# for the pod CIDR) was REJECTED: it leaves the human login broken, and makes qBittorrent's auth
|
|
# depend on network policy staying correct forever. The WebUI keeps demanding a password; that
|
|
# password is simply now known, fixed, and reproducible from git.
|
|
postStart:
|
|
exec:
|
|
command:
|
|
- /bin/bash
|
|
- -c
|
|
- |
|
|
# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the
|
|
# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.
|
|
#
|
|
# THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.
|
|
# apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over
|
|
# this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;
|
|
# enforced by render.py's assert_substitution_safe) makes such a form correct ON THE
|
|
# CLUSTER — but only there:
|
|
# envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl
|
|
# apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during
|
|
# development) keeps `$$`, which bash expands to its own PID. That is not a hypothetical;
|
|
# see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`
|
|
# are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY
|
|
# however it is run.
|
|
set -u
|
|
CONF=/config/qBittorrent/qBittorrent.conf
|
|
SVC=/run/service/svc-qbittorrent
|
|
# The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this
|
|
# ServiceDefinition — which is the single place the renderer takes the containerPort, the
|
|
# Service port and the HTTPRoute backend port from — and the image exposes no WEBUI_PORT
|
|
# env var for a lookup to read anyway.
|
|
# It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and
|
|
# that cost real time: run without envsubst, bash read the doubled dollar as its own PID,
|
|
# so PORT became "<pid>{WEBUI_PORT:-8080}" — the brace never expanded, the digits never
|
|
# matched. Every curl below then got an unparseable URL, returned nothing, and the hook
|
|
# logged "the WebUI rejected the durable
|
|
# credential" — having never actually asked it. That MISDIAGNOSIS is preserved in this
|
|
# service's own marker log at 2026-09-13T05:35Z (three "rejected" lines, then GAVE UP,
|
|
# against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.
|
|
PORT=8080
|
|
# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` — so a hook
|
|
# that only echoes is effectively silent (gitea's hits the same wall and writes a marker
|
|
# file for it). Every outcome therefore also lands, timestamped, on the persistent config
|
|
# volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the
|
|
# credential from git was asserted and accepted on this start.
|
|
log() {
|
|
echo "[qbt-webui-cred] $*"
|
|
printf '%s [qbt-webui-cred] %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" \
|
|
>> /config/qbittorrent-webui-cred.log 2>/dev/null || true
|
|
}
|
|
|
|
# (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC
|
|
# before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of
|
|
# racing to create one it would then not overwrite.
|
|
n=0
|
|
while [ ! -s "$CONF" ] && [ "$n" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done
|
|
|
|
waited=0
|
|
attempt=0
|
|
while [ "$attempt" -lt 3 ]; do
|
|
attempt=$((attempt + 1))
|
|
|
|
# (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),
|
|
# 10 = rewritten, anything else = could not write.
|
|
python3 - "$CONF" <<'PY'
|
|
import base64, hashlib, hmac, os, stat, sys
|
|
|
|
conf = sys.argv[1]
|
|
user = os.environ.get("QBT_WEBUI_USERNAME", "")
|
|
pw = os.environ.get("QBT_WEBUI_PASSWORD", "")
|
|
if not user or not pw:
|
|
sys.stderr.write("[qbt-webui-cred] credential env is empty - refusing to touch the conf\n")
|
|
raise SystemExit(1)
|
|
|
|
# qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000
|
|
# iterations, 16-byte random salt, 64-byte digest, stored as
|
|
# WebUI\Password_PBKDF2="@ByteArray(<base64 salt>:<base64 digest>)"
|
|
# Get any one of them wrong and the app simply rejects every login.
|
|
ITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16
|
|
UKEY, PKEY, SECTION = "WebUI\\Username", "WebUI\\Password_PBKDF2", "[Preferences]"
|
|
|
|
def matches(raw):
|
|
"""True iff the stored digest is one OUR password produces (salt is read from it)."""
|
|
v = raw.strip().strip('"')
|
|
if v.startswith("@ByteArray(") and v.endswith(")"):
|
|
v = v[len("@ByteArray("):-1]
|
|
try:
|
|
salt_b64, digest_b64 = v.split(":", 1)
|
|
salt = base64.b64decode(salt_b64, validate=True)
|
|
digest = base64.b64decode(digest_b64, validate=True)
|
|
except Exception:
|
|
return False
|
|
if not salt or not digest:
|
|
return False
|
|
return hmac.compare_digest(
|
|
hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, len(digest)), digest)
|
|
|
|
try:
|
|
lines = open(conf, encoding="utf-8", errors="replace").read().splitlines()
|
|
except FileNotFoundError:
|
|
lines = []
|
|
|
|
def value_of(key):
|
|
for i, line in enumerate(lines):
|
|
if line.startswith(key + "="):
|
|
return i, line.split("=", 1)[1]
|
|
return -1, None
|
|
|
|
ui, uval = value_of(UKEY)
|
|
pi, pval = value_of(PKEY)
|
|
if uval == user and pval is not None and matches(pval):
|
|
raise SystemExit(0) # converged - leave the file (and its salt) alone
|
|
|
|
salt = os.urandom(SALT_LEN)
|
|
digest = hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, DIGEST_LEN)
|
|
stored = base64.b64encode(salt).decode() + ":" + base64.b64encode(digest).decode()
|
|
desired = [UKEY + "=" + user, PKEY + '="@ByteArray(' + stored + ')"']
|
|
|
|
out = [line for i, line in enumerate(lines) if i not in (ui, pi)]
|
|
if SECTION in out:
|
|
at = out.index(SECTION) + 1
|
|
else:
|
|
if out and out[-1].strip():
|
|
out.append("")
|
|
out.append(SECTION)
|
|
at = len(out)
|
|
out[at:at] = desired
|
|
|
|
try:
|
|
os.makedirs(os.path.dirname(conf) or ".", exist_ok=True)
|
|
tmp = conf + ".webui-cred.new"
|
|
with open(tmp, "w", encoding="utf-8") as fh:
|
|
fh.write("\n".join(out) + "\n")
|
|
fh.flush()
|
|
os.fsync(fh.fileno())
|
|
# This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would
|
|
# cost the app the ability to save its own settings, so carry the ownership over.
|
|
try:
|
|
st = os.stat(conf)
|
|
os.chown(tmp, st.st_uid, st.st_gid)
|
|
os.chmod(tmp, stat.S_IMODE(st.st_mode))
|
|
except FileNotFoundError:
|
|
os.chown(tmp, int(os.environ.get("PUID", "1000")),
|
|
int(os.environ.get("PGID", "1000")))
|
|
os.chmod(tmp, 0o664)
|
|
os.replace(tmp, conf) # atomic: the app never reads a half-written conf
|
|
except OSError as exc:
|
|
sys.stderr.write("[qbt-webui-cred] cannot write %s: %s\n" % (conf, exc))
|
|
raise SystemExit(1)
|
|
raise SystemExit(10)
|
|
PY
|
|
rc=$?
|
|
|
|
if [ "$rc" = 0 ]; then
|
|
log "conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)"
|
|
elif [ "$rc" = 10 ]; then
|
|
log "asserted WebUI Username + Password_PBKDF2 into $CONF"
|
|
# postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY
|
|
# have read the old, credential-less conf and be serving a random temporary password.
|
|
# Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.
|
|
# Nothing to do when the service has not started yet - it will read the new file.
|
|
# Its own counter, NOT the shared `waited` API budget: sharing one counter is how the
|
|
# 2026-09-13 failure spent the entire verification budget before the first login and
|
|
# then fired attempts 2 and 3 with no wait left at all.
|
|
sv=0
|
|
while [ "$sv" -lt 60 ] && [ "$(s6-svstat -u "$SVC" 2>/dev/null)" != "true" ]; do
|
|
sv=$((sv + 1)); sleep 1
|
|
done
|
|
if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
|
|
log "qbittorrent-nox was already running - restarting it to load the credential"
|
|
s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)"
|
|
sleep 3
|
|
fi
|
|
else
|
|
log "FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep" >&2
|
|
log "minting a random password every start and BOTH arrs will break. Fix and restart." >&2
|
|
exit 0
|
|
fi
|
|
|
|
# (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL
|
|
# (shared ~120s budget across attempts), then spend exactly ONE login per attempt -
|
|
# at most 3 rejected logins, which stays under qBittorrent's default ban threshold.
|
|
#
|
|
# "THE API NEVER ANSWERED" IS A DIFFERENT FAULT FROM "THE PASSWORD WAS REJECTED", and
|
|
# conflating them is what made this hook lie once already (see the PORT comment). An
|
|
# unreachable WebUI is reported as unreachable, and the hook stops rather than spending
|
|
# its remaining login attempts to produce two more identical wrong diagnoses.
|
|
api_ok=0
|
|
while [ "$waited" -lt 120 ]; do
|
|
code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' "http://127.0.0.1:$PORT/" 2>/dev/null)
|
|
if [ -n "$code" ] && [ "$code" != "000" ]; then api_ok=1; break; fi
|
|
waited=$((waited + 1)); sleep 1
|
|
done
|
|
if [ "$api_ok" != 1 ]; then
|
|
log "the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the" >&2
|
|
log "credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was" >&2
|
|
log "never submitted. Do not rotate the credential on the strength of this line." >&2
|
|
exit 0
|
|
fi
|
|
body=$(curl -sS -m 10 \
|
|
--data-urlencode "username=$QBT_WEBUI_USERNAME" \
|
|
--data-urlencode "password=$QBT_WEBUI_PASSWORD" \
|
|
"http://127.0.0.1:$PORT/api/v2/auth/login" 2>/dev/null)
|
|
if [ "$body" = "Ok." ]; then
|
|
log "VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'"
|
|
exit 0
|
|
fi
|
|
log "WebUI rejected the durable credential (login said '$body')" >&2
|
|
# A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean
|
|
# the running qbittorrent-nox is serving something it read from an EARLIER conf, so
|
|
# rewriting the same bytes on the next attempt would change nothing. The remedy is to
|
|
# make the process RE-READ the file, so restart the supervised service and let the next
|
|
# attempt re-verify. Without this the retry loop could never change its own outcome.
|
|
if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
|
|
log "restarting qbittorrent-nox so it re-reads $CONF, then re-verifying" >&2
|
|
s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)" >&2
|
|
sleep 3
|
|
fi
|
|
done
|
|
|
|
log "GAVE UP after $attempt attempts: the WebUI does not accept the credential from git." >&2
|
|
# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container
|
|
# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.
|
|
# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the
|
|
# app, and the arrs' own download-client health check surfaces a broken credential.
|
|
exit 0
|
|
podSecurityContext:
|
|
# For the LONGHORN config PVC only, and for the same reason radarr/sonarr set it: fsGroup is a NO-OP
|
|
# on the NFS volume (the in-tree plugin does no ownership management), but on a Longhorn filesystem
|
|
# the kubelet applies it, so /config becomes group-writable by the now-uid-1000 process without
|
|
# depending on the image chowning it. (This image DOES chown: init-qbittorrent-config runs
|
|
# `lsiown -R abc:abc /config`. Belt and braces — the live /config was 911:911 before this change,
|
|
# and losing write access to it would lose the WebUI password hash and the session state.)
|
|
fsGroup: 3000
|
|
exposure: internal # LAN-only web UI at qbittorrent.${BASE_DOMAIN} (docs/13 row 13; R-DNS/R-NET)
|
|
egress: vpn # ALL public egress via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
|
|
storage:
|
|
- name: config
|
|
size: 5Gi
|
|
class: longhorn # settings/categories/session on 3-replica Longhorn; DR-critical (docs/04; R-STATE-02)
|
|
mountPath: /config # THE APP WRITES /config, NOT the renderer default /data/config.
|
|
# LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
|
|
# /config), so without this the Longhorn PVC mounted at /data/config sat
|
|
# holding nothing but lost+found while the app wrote its entire
|
|
# configuration to the container's EPHEMERAL writable layer — destroyed on
|
|
# every pod recreation: restart, reschedule, image bump. Measured in-cluster
|
|
# 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
|
|
# dev=2053 on the node's own disk with [Sentry asp config.xml logs].
|
|
# This is the SAME failure catalog/plex.yaml:54 already records fixing.
|
|
# The live /config contents were copied onto each PVC before this landed,
|
|
# so API keys and credentials other apps reference were preserved.
|
|
# THE DOWNLOADS AREA — ONE mount of the SAME export sonarr/radarr/plex already consume
|
|
# (192.168.2.102:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
|
|
# /downloads, which is precisely where this app is ALREADY configured to write.
|
|
#
|
|
# WHY subPath: downloads AND NOT THE EXPORT ROOT AT /media (the shape sonarr/radarr use):
|
|
# * LEAST PRIVILEGE, AND IT COSTS NOTHING HERE. A download client has no business writing the
|
|
# LIBRARY. The arrs need the root mount because THEY hardlink downloads/ -> tv/ or movies/ and
|
|
# link() is a same-MOUNT operation (EXDEV otherwise) — their smallest common ancestor genuinely
|
|
# IS the export root. qBittorrent performs no such cross-directory link: it writes into the
|
|
# downloads tree and nothing else, so narrowing to that subtree removes movies/, tv/ and photos/
|
|
# from this pod's mount namespace without removing any operation it actually performs.
|
|
# * HARDLINKS STILL WORK, because they are not done here. The arrs still see ONE mount containing
|
|
# both /media/downloads and /media/tv, so their link() stays intra-mount. Proven live 2026-09-12:
|
|
# a file created from this pod under /downloads was hardlinked from the sonarr pod between
|
|
# /media/downloads/... and /media/tv/... with st_nlink 1 -> 2 and no EXDEV.
|
|
# * IT IS ONE MOUNT, so the L0 test `bulk-tier-distinct-from-replicated` assertion (6) — no
|
|
# workload mounts one export twice — holds. That assertion is about TWO mounts inside ONE pod;
|
|
# a single subPath mount is not the EXDEV trap, two are.
|
|
# * The narrowing is a CLIENT/kubelet guarantee, stated honestly: the NFS server still exports the
|
|
# whole dataset to this node, so the containment is the mount namespace, not the server's ACLs —
|
|
# the same class of caveat docs/BULK-NFS-STORAGE.md §5 records for plex's readOnly.
|
|
#
|
|
# NOT readOnly — this is the one workload whose whole job is WRITING here.
|
|
#
|
|
# KNOWN, DELIBERATE SIDE EFFECT ON SHARED STATE: the image's init-qbittorrent-config runs
|
|
# `lsiown abc:abc /downloads` (non-recursive) whenever /downloads is a mount, so starting this pod
|
|
# chgrps /mnt/Storage/plex/downloads from 1000:1000 to 1000:3000 on the SERVER. Benign and verified:
|
|
# the directory is mode 0755, whose group bits (r-x) are identical to its other bits, so no consumer
|
|
# — k3s or legacy — gains or loses any access. Recorded because it is a write to state shared with
|
|
# non-cluster consumers, which should never be a surprise.
|
|
#
|
|
# PATH AGREEMENT WITH THE ARRS IS STILL *NOT* CLOSED — see DECISIONS-LOG 2026-09-12 "Remote Path
|
|
# Mappings". This pod calls the finished file /downloads/<name>; sonarr/radarr call the same bytes
|
|
# /media/downloads/<name>. Same inode, different absolute path, so Completed Download Handling needs
|
|
# ONE Remote Path Mapping per arr (remote host `qbittorrent.qbittorrent.svc.cluster.local`, remote
|
|
# path /downloads/, local path /media/downloads/).
|
|
#
|
|
# THOSE MAPPINGS ARE NOW CONFIGURED AND VERIFIED ON PROD (2026-09-12), together with the qBittorrent
|
|
# download client itself — which did not exist in EITHER arr until then (both
|
|
# `GET /api/v3/downloadclient` returned `[]`, so completed-download handling had never been wired up
|
|
# at all). `downloadclient/testall` reports isValid=true in both, and Sonarr's queue was observed
|
|
# reporting outputPath=/downloads/... with the mapping deleted and /media/downloads/... with it
|
|
# present — same torrent, same client.
|
|
#
|
|
# BUT THEY LIVE IN THE ARRS' OWN DATABASES (app state on their config PVCs), NOT IN GIT: a rebuild
|
|
# from git alone does NOT restore them and no gate here notices their absence. The alternative —
|
|
# repointing qBittorrent's SavePath at /media/downloads and mounting this subPath there instead —
|
|
# buys path agreement at the cost of editing THIS app's config PVC, which is also app state; it was
|
|
# not chosen because it changes a running download client's save path, which the deferral above
|
|
# already shows is easy to get wrong. It is nonetheless the RECOMMENDED end state (it makes the
|
|
# mappings unnecessary and puts the mountPath half of the fix in git) and is awaiting an operator
|
|
# decision, not a technical blocker.
|
|
#
|
|
# RELATED OPEN BLOCKER: this app has no durable WebUI credential (no WebUI\Username /
|
|
# WebUI\Password_PBKDF2 in qBittorrent.conf), so v5 issues a NEW temporary `admin` password on every
|
|
# start. The arrs' saved download-client password therefore goes stale on every pod restart. See the
|
|
# same DECISIONS-LOG entry for the two fixes, neither of which was taken unilaterally.
|
|
- name: downloads
|
|
class: bulk
|
|
export: /mnt/Storage/plex # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER} — the SAME export
|
|
# plex (ro) and sonarr/radarr (rw) mount; one dataset, one set of inodes
|
|
subPath: downloads # the handoff subtree ONLY: no movies/, no tv/, no photos/
|
|
mountPath: /downloads # matches the app's EXISTING Session\DefaultSavePath / TempPath, so no
|
|
# change to config-PVC app state is needed to make this take effect
|
|
size: 9Ti # ADVISORY ONLY on a static NFS PV (the ZFS pool is the real limit): the
|
|
# dataset's 8.4 T used rounded up, stated the same way plex/sonarr/radarr
|
|
# state it so `kubectl get pv` reports something true
|
|
dependencies: [] # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02).
|
|
# STILL EMPTY WITH THE NFS MOUNT ABOVE, AND THAT IS CORRECT, NOT AN
|
|
# OVERSIGHT. An NFS PersistentVolume is mounted by the NODE KERNEL in the
|
|
# HOST netns, not by this pod: there is no pod-side socket to 2049, so no
|
|
# CiliumNetworkPolicy can permit or deny it and none is needed. Verified by
|
|
# construction — sonarr and radarr have mounted this same export read-write
|
|
# for weeks and neither has any CNP naming the NFS server or port 2049
|
|
# (grep apps/wave-1/sonarr.yaml: none). The SAME fact is what keeps the VPN
|
|
# egress gateway off this traffic (its selector is a podSelector), and
|
|
# 192.168.2.102 is inside the 192.168.0.0/16 excludedCIDR besides — see
|
|
# platform/vpn-egress/egress-steering.yaml, "BULK NFS".
|
|
ingressRoutes:
|
|
- host: qbittorrent.${BASE_DOMAIN}
|
|
port: 8080 # qBittorrent WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
|
|
auth:
|
|
mode: forward-auth # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)
|
|
groups:
|
|
- media-users # authenticated AND in an authorized group (R-IDP-06)
|
|
# See catalog/jackett.yaml for the full rationale: --allowed-group is repeatable and OR-ed, so
|
|
# platform-admins here is a superset, not a widening (that group already carries cluster-admin
|
|
# via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
|
|
# immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
|
|
- platform-admins # operators of these services must not be locked out of them
|
|
update:
|
|
channel: stable # R-UPD-01
|
|
range: ">=5.0.0 <6.0.0" # stay on the qBittorrent 5.x stable line
|
|
cadence: "0 4 * * *"
|
|
autoRollbackOnFailedHealth: true # R-UPD-03
|
|
observability:
|
|
metrics: true
|
|
slis:
|
|
- up # download client down -> alert (docs/10; R-OBS-01)
|
|
backup:
|
|
policy: daily-offcluster # config (categories/settings) backed up off-cluster (docs/04; R-STATE-03)
|