jakman85andClaude Opus 5
99560c2d22
PR #28 fixed the cookie collision, the 5-minute session and the missing metrics
listener in one renderer function, and deliberately declined `--api-route`
because the config model could not express the difference between two apps.
This adds the hook it was missing and sets the flag only where a measurement
says to.
THE DEFECT. oauth2-proxy answers an unauthenticated request with a 302 to
Keycloak. Its AJAX guard would send 401 instead, but it fires only on an EXACT
`Accept: application/json`, which these apps' JavaScript does not consistently
send. So a browser XHR gets a 302, FOLLOWS IT INVISIBLY, has it satisfied
silently by the live SSO session, and resolves 200 carrying the app's HTML where
the JavaScript expected JSON. On a write it is worse: a browser following a 302
re-issues a POST as a GET and DROPS THE BODY, so the operator's change is
discarded and the UI reports success.
MEASURED, on two apps and two different paths -- which is the whole argument:
jackett 2026-09-30: 33 x 302 vs 2 x 401 on /api/ paths, and 4 of 6 POSTs
answered by the proxy and never reaching Jackett (upstream `-`).
radarr 2026-10-01, 23h after PR #28 deployed (pod radarr-7d55cbffb9-42hmh,
0 restarts): 232 unauthenticated requests in the pod's whole life and
ALL 232 are the same call -- POST /signalr/messages/negotiate from a
browser, 302, upstream `-`. A left-open tab whose session the cookie
rename orphaned, polling forever, silently redirected every time and
NEVER re-prompting. The reported symptom, still in production.
So radarr's broken path is /signalr/ and jackett's is /api/, and PIHOLE SERVES
NO /api/ AT ALL. One function renders all five sidecars, so the cheap fix is a
constant -- and a constant applies one app's measurement to four apps nobody
measured. `auth` was `additionalProperties: false`, so there was no per-app
hook. Adding one named, documented key is the change; the block stays closed.
schemas/servicedefinition.schema.json 1.8.0 -> 1.9.0: optional
`auth.apiRoutes`, a list (radarr needs two, and --api-route is natively
repeatable) of anchored path regexes. Plus the shape invariant that a
non-empty list requires mode: forward-auth, since no sidecar is rendered
for oidc/native/none and declaring it there would be a SILENT NO-OP.
catalog/render.py emits one --api-route per entry, in order, and RAISES on
an unanchored entry. The flag is SINGULAR -- verified against the pinned
v7.8.1 binary's own --help; --api-routes does not exist and the proxy
refuses to start with it.
catalog/{jackett,radarr}.yaml the two measured apps.
catalog/{pihole,sonarr,qbittorrent}.yaml comments recording that they get
NOTHING, and why. Sonarr and qBittorrent were measured (0 unauthenticated
requests in 23h) and left alone: that is evidence of no TRAFFIC, not of no
need, and sonarr being nearly radarr's codebase is what makes the guess
tempting and still a guess.
apps/wave-0/jackett.yaml, apps/wave-1/radarr.yaml regenerated. Only TWO
goldens change; pihole/qbittorrent/sonarr are byte-identical to main,
which is the per-app property proving itself on the real tree.
BREADTH IS THE DANGEROUS DIRECTION AND IT IS NOT SYMMETRIC. A missing
--api-route is a silent 302. A too-broad one answers a plain PAGE LOAD with
401, so a logged-out operator gets a bare error page and is never offered a
sign-in -- the service looks permanently broken. Each entry must therefore
match `^\^/[A-Za-z0-9._~/-]+(\.\*|\$)?$`: anchored at ^/ then literal path
characters. `^/`, `^/.*`, `.*`, `/api/` and `^/api/|^/` are all rejected, all
five provoked. Alternation is refused rather than supported because the flag is
repeatable, so one entry per route costs nothing.
NEW GATE CHECK: conformance/tests/l0-static/oauth2-proxy-api-routes-per-app
(whole-repo scope, so ci/l0.sh's --include-scope union runs it on every
invocation; no bespoke stage, which whole-repo-invariants-wired would fail).
It is a SIBLING of oauth2-proxy-cookie-names-distinct, not an extension of it:
that check's subject is cookie IDENTITY and shares neither inputs nor failure
mode with this, so bundling them would make its name a lie and let a future
api-route decision be expressed only by weakening a COOKIE check.
The check exists for one reason the other gates cannot cover: render-golden
compares the render to the goldens, so a RENDERER CONSTANT plus regenerated
goldens AGREE WITH EACH OTHER while both being wrong. Provoked: the regen exits
0 -- render-golden is happy -- and this check exits 1 with 15 failures naming
pihole, qbittorrent, radarr and sonarr individually. It also asserts the
anchoring pattern is the SAME in the schema and the renderer, because three
surfaces guard this and if they drift the loosest wins silently.
Tested in both directions on throwaway trees outside the repo: compliant tree
rc 0 / 43 assertions / 6 sidecars; pre-change tree rc 1 / 11 failures; and four
targeted mutations (hand-edited golden, widened regex, renderer constant,
deleted schema pattern) each firing their own assertion class and nothing else.
ALSO FOLDED IN (coordinator's request; defect is in the file I own). _selftest()
reached _cookie_name() through a bare `next(genexpr)` with NO default, so
deleting the --cookie-name emission -- the exact mutation that assertion exists
to catch -- raised StopIteration, which escaped _selftest() BEFORE its collected
`SELFTEST FAIL:` lines were printed. Right verdict, zero diagnostics, and it
masked every other assertion the same mutation broke (found by Desmond Oyelaran
while wiring the selftest into the render-golden stage; his PR #33 is not in this
base -- origin/main @ cdf3e14 drives the selftest from no stage, checked). Fixed
with `next(..., None)` plus an explicit check(), so a missing flag reports as a
NAMED assertion. Provoking it exposed three more instances of the same class in
the same block -- `side = fa_containers[1]`, `side["env"][0]`, and the fa_svc
lookup all raise rather than assert when the sidecar is absent -- so all four are
now None-tolerant, and the two fixtures added for --api-route were written that
way from the start. Measured on throwaway copies:
--cookie-name deleted: before rc 1 / 0 FAIL lines / StopIteration
after rc 1 / 3 FAIL lines / no traceback
sidecar not appended: before rc 1 / 0 FAIL lines / IndexError
after rc 1 / 20 FAIL lines / no traceback
Goldens are unchanged by this part; it touches _selftest() only.
NOT verified, in full in the handoff entry: no browser round-trip by anyone for
this flag -- that a 401 REPAIRS the operator's experience is reasoned from HTTP
semantics, not watched. `^/api/` on radarr is PR #30's measurement of radarr's
pre-fix pod, not mine; the current pod has zero /api/ traffic. Sonarr and
qBittorrent are UNMEASURED, not cleared. --cookie-refresh is still unexercised
(0 AuthSuccess on all five pods). Nothing was applied; the live pods are still
the pre-change spec.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
468 lines
31 KiB
YAML
468 lines
31 KiB
YAML
# ServiceDefinition: qbittorrent
|
|
# Source of truth: spec docs/13-service-catalog.md row 13; docs/03-networking-isolation-and-egress.md;
|
|
# docs/15-identity-and-access.md; decisions/adr-0015 (VPN egress steering).
|
|
# internal download client | egress VPN + kill-switch | config on Longhorn | auth forward-auth
|
|
# (bare web UI, no native OIDC).
|
|
#
|
|
# BULK DOWNLOADS TIER — DELIVERED 2026-09-12 (the deferral below is CLOSED; kept for the record).
|
|
# This entry used to ship config-only, with a comment promising to attach the downloads volume "once
|
|
# OQ-2 is resolved". OQ-2 was resolved for STATIC pre-existing datasets (schema 1.5.0 / renderer
|
|
# `class: bulk` + `export` -> a static, pre-bound NFS PV), and plex/sonarr/radarr/nextcloud all took
|
|
# their media mounts — but qBittorrent's was never added, so the DOWNLOAD CLIENT was the one member
|
|
# of the chain with no media volume at all. Measured on prod 2026-09-12:
|
|
# Session\DefaultSavePath=/downloads/ Session\TempPath=/downloads/incomplete/
|
|
# Downloads\SavePath=/downloads/ `ls /downloads` -> No such file or directory
|
|
# i.e. the app was configured to write a path that DID NOT EXIST, in a container with no volume there.
|
|
# Anything it did manage to create would have been invisible to sonarr/radarr, lost on every pod
|
|
# recreation, and filling the NODE's ephemeral disk in the meantime. The `downloads` entry under
|
|
# storage: below closes that. See docs/BULK-NFS-STORAGE.md and the bulk-tier L0 conformance test.
|
|
#
|
|
# VPN-class (docs/13 row 13; docs/03 "VPN-class set"): ALL of this pod's public-internet egress
|
|
# (peer/tracker/DHT traffic) MUST leave via the node PIA tunnel, never the WAN. `egress: vpn` renders
|
|
# the homelab.keithjackson.dev/egress-class=vpn pod label the shared CiliumEgressGatewayPolicy
|
|
# (platform/vpn-egress/egress-steering.yaml) selects on to steer + kill-switch the traffic OUTSIDE the
|
|
# workload (R-NET-05/06/07). No independent egress path; fail-closed if tun0 is down.
|
|
#
|
|
# Inbound reachability (<- sonarr, <- radarr) is an EAST-WEST allow rendered on the arr side; being
|
|
# VPN-class governs egress, not intra-cluster reachability. This service initiates nothing east-west, so
|
|
# dependencies is empty (reaches only DNS + VPN internet).
|
|
# Validates against schemas/servicedefinition.schema.json.
|
|
name: qbittorrent
|
|
kind: container
|
|
source:
|
|
image: lscr.io/linuxserver/qbittorrent:5.1.2 # pinned stable tag (R-UPD-01); update.range governs advancement
|
|
# LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
|
|
# LOAD-BEARING FOR THE BULK MOUNT, not cosmetic — exactly as on sonarr/radarr. Measured on prod
|
|
# 2026-09-12 BEFORE this change: `id` inside the container reported the qbittorrent-nox process as
|
|
# uid=911 gid=911, and /mnt/Storage/plex/downloads is 0755 owned 1000:1000. 0755 grants write to the
|
|
# OWNER ONLY, so uid 911 would have mounted the export cleanly and then failed every write with
|
|
# EACCES — the "mounts fine, cannot write" failure the sonarr entry warns about at length.
|
|
env:
|
|
- name: PUID
|
|
value: "1000" # owner uid of /mnt/Storage/plex/downloads (and of every dir under the export)
|
|
- name: PGID
|
|
value: "3000" # the export's dominant group, matching sonarr/radarr: files this writes stay
|
|
# readable to the EXISTING non-k3s consumers of the same dataset, and land
|
|
# group-owned the same way the arrs' own writes do
|
|
# --- THE DURABLE WebUI CREDENTIAL (consumed by the postStart hook below) -------------------
|
|
# Measured on prod 2026-09-13: /config/qBittorrent/qBittorrent.conf held NEITHER `WebUI\Username`
|
|
# NOR `WebUI\Password_PBKDF2`. qBittorrent 5.x reads that as "no credential configured" and mints a
|
|
# NEW random `admin` password on EVERY process start, announcing it ONLY on stdout ("A temporary
|
|
# password is provided for this session: ..."). Two real consequences, not hypotheticals:
|
|
# * Sonarr and Radarr store a download-client username/password in their OWN databases. Both
|
|
# tested valid against the then-current temporary password; both would have started failing at
|
|
# the NEXT qbittorrent pod restart, silently breaking Completed Download Handling (the arrs
|
|
# report a failed client, not a failed import).
|
|
# * The human operator had no stable login: every restart meant reading the pod log.
|
|
# Provided as apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml (SOPS/age, decrypted by Flux) —
|
|
# same shape and naming as gitea-bootstrap, so `render.py --emit-waves` places it in this wave.
|
|
- name: QBT_WEBUI_USERNAME
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: qbittorrent-bootstrap
|
|
key: webui-username
|
|
- name: QBT_WEBUI_PASSWORD
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: qbittorrent-bootstrap
|
|
key: webui-password
|
|
lifecycle:
|
|
# LEVEL-TRIGGERED CREDENTIAL ASSERTION, run on EVERY container start.
|
|
#
|
|
# WHY A postStart HOOK AND NOT A CronJob RECONCILER: the repo's idiom for "state Flux cannot reach
|
|
# directly" is the level-triggered CronJob (platform/keycloak/client-secret-reconciler.yaml,
|
|
# apps/wave-0/nextcloud-db.yaml). That idiom's PRINCIPLE — desired state lives in git and is
|
|
# RE-ASSERTED unconditionally, never set once — is exactly what is applied here, but the trigger is
|
|
# the container start rather than a clock, because the failure this closes IS a container start:
|
|
# * The credential is read by qbittorrent-nox ONCE, at process start, from a file on the config
|
|
# PVC. A 15-minute cron would therefore leave a 0-15 minute window after every restart, PVC
|
|
# restore or rebuild in which the app serves a random password and both arrs are broken. The
|
|
# hook has no such window: it runs before/with the very start that would otherwise regenerate.
|
|
# * A CronJob would also have to reach the conf from OUTSIDE the pod — mounting an RWO Longhorn
|
|
# PVC that the app already holds, or `kubectl exec`-ing with RBAC it would need granting — and
|
|
# then edit a file UNDER the running process, whose own save-on-exit can overwrite the edit.
|
|
# In-container, before the service starts, none of that exists.
|
|
# * It needs no ServiceAccount, no RBAC, no NetworkPolicy and no API token: strictly less
|
|
# privilege than the CronJob alternative (R-SEC/least-privilege).
|
|
# The gitea entry's postStart is the same pattern (idempotent bootstrap of a credential the app
|
|
# keeps in its own state), and the schema documents this hook for exactly that use.
|
|
#
|
|
# AND WHY NOT AN initContainer, which is the genuinely tempting third option: it would run strictly
|
|
# BEFORE qbittorrent-nox, removing the race the `s6-svc -r` below exists to cover, and its stdout
|
|
# would reach `kubectl logs -c <init>` instead of the kubelet. Two reasons it loses here:
|
|
# * It CANNOT do step (3) — the ground-truth login that proves the WebUI actually accepts the
|
|
# credential — because the WebUI is not running yet. That assertion is the most valuable line
|
|
# in this whole mechanism: it is what turns "we wrote a file" into "the credential works", and
|
|
# writing the right PBKDF2 bytes is precisely the part that is easy to get subtly wrong.
|
|
# * The ServiceDefinition schema (1.5.0) has `lifecycle` but NO initContainers field, so it would
|
|
# mean a schema version bump plus a renderer change — a blast radius across every service's
|
|
# golden — to buy something strictly weaker than what the hook already does.
|
|
# The honest summary: postStart trades a short, self-healing start-up race (detected and closed by
|
|
# the s6 restart + re-verify loop) for the ability to VERIFY. That is the right trade.
|
|
#
|
|
# DURABILITY, the whole point — all three paths end with the SAME credential:
|
|
# pod restart -> conf on the PVC already holds it; the hook verifies and writes NOTHING.
|
|
# config-PVC restore from an older backup -> restore requires the pod to be recreated, so the
|
|
# hook runs and re-asserts over whatever the snapshot contained.
|
|
# rebuild from git -> empty PVC; the image copies /defaults/qBittorrent.conf, the hook writes the
|
|
# credential into it before the service is serving. Nothing outside git.
|
|
#
|
|
# WHAT IT DOES NOT DO: it never weakens authentication. `WebUI\AuthSubnetWhitelist` (bypassing auth
|
|
# for the pod CIDR) was REJECTED: it leaves the human login broken, and makes qBittorrent's auth
|
|
# depend on network policy staying correct forever. The WebUI keeps demanding a password; that
|
|
# password is simply now known, fixed, and reproducible from git.
|
|
postStart:
|
|
exec:
|
|
command:
|
|
- /bin/bash
|
|
- -c
|
|
- |
|
|
# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the
|
|
# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.
|
|
#
|
|
# THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.
|
|
# apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over
|
|
# this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;
|
|
# enforced by render.py's assert_substitution_safe) makes such a form correct ON THE
|
|
# CLUSTER — but only there:
|
|
# envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl
|
|
# apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during
|
|
# development) keeps the doubled dollar, which bash reads as its own PID. Not hypothetical;
|
|
# see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`
|
|
# are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY
|
|
# however it is run.
|
|
set -u
|
|
CONF=/config/qBittorrent/qBittorrent.conf
|
|
SVC=/run/service/svc-qbittorrent
|
|
# The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this
|
|
# ServiceDefinition — which is the single place the renderer takes the containerPort, the
|
|
# Service port and the HTTPRoute backend port from — and the image exposes no WEBUI_PORT
|
|
# env var for a lookup to read anyway.
|
|
# It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and
|
|
# that cost real time: run without envsubst, bash read the doubled dollar as its own PID,
|
|
# so PORT became "<pid>{WEBUI_PORT:-8080}" — the brace never expanded, the digits never
|
|
# matched. Every curl below then got an unparseable URL, returned nothing, and the hook
|
|
# logged "the WebUI rejected the durable
|
|
# credential" — having never actually asked it. That MISDIAGNOSIS is preserved in this
|
|
# service's own marker log at 2026-09-13T05:35Z (three "rejected" lines, then GAVE UP,
|
|
# against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.
|
|
PORT=8080
|
|
# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` — so a hook
|
|
# that only echoes is effectively silent (gitea's hits the same wall and writes a marker
|
|
# file for it). Every outcome therefore also lands, timestamped, on the persistent config
|
|
# volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the
|
|
# credential from git was asserted and accepted on this start.
|
|
log() {
|
|
echo "[qbt-webui-cred] $*"
|
|
printf '%s [qbt-webui-cred] %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" \
|
|
>> /config/qbittorrent-webui-cred.log 2>/dev/null || true
|
|
}
|
|
|
|
# (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC
|
|
# before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of
|
|
# racing to create one it would then not overwrite.
|
|
n=0
|
|
while [ ! -s "$CONF" ] && [ "$n" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done
|
|
|
|
waited=0
|
|
attempt=0
|
|
while [ "$attempt" -lt 3 ]; do
|
|
attempt=$((attempt + 1))
|
|
|
|
# (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),
|
|
# 10 = rewritten, anything else = could not write.
|
|
python3 - "$CONF" <<'PY'
|
|
import base64, hashlib, hmac, os, stat, sys
|
|
|
|
conf = sys.argv[1]
|
|
user = os.environ.get("QBT_WEBUI_USERNAME", "")
|
|
pw = os.environ.get("QBT_WEBUI_PASSWORD", "")
|
|
if not user or not pw:
|
|
sys.stderr.write("[qbt-webui-cred] credential env is empty - refusing to touch the conf\n")
|
|
raise SystemExit(1)
|
|
|
|
# qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000
|
|
# iterations, 16-byte random salt, 64-byte digest, stored as
|
|
# WebUI\Password_PBKDF2="@ByteArray(<base64 salt>:<base64 digest>)"
|
|
# Get any one of them wrong and the app simply rejects every login.
|
|
ITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16
|
|
UKEY, PKEY, SECTION = "WebUI\\Username", "WebUI\\Password_PBKDF2", "[Preferences]"
|
|
|
|
def matches(raw):
|
|
"""True iff the stored digest is one OUR password produces (salt is read from it)."""
|
|
v = raw.strip().strip('"')
|
|
if v.startswith("@ByteArray(") and v.endswith(")"):
|
|
v = v[len("@ByteArray("):-1]
|
|
try:
|
|
salt_b64, digest_b64 = v.split(":", 1)
|
|
salt = base64.b64decode(salt_b64, validate=True)
|
|
digest = base64.b64decode(digest_b64, validate=True)
|
|
except Exception:
|
|
return False
|
|
if not salt or not digest:
|
|
return False
|
|
return hmac.compare_digest(
|
|
hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, len(digest)), digest)
|
|
|
|
try:
|
|
lines = open(conf, encoding="utf-8", errors="replace").read().splitlines()
|
|
except FileNotFoundError:
|
|
lines = []
|
|
|
|
def value_of(key):
|
|
for i, line in enumerate(lines):
|
|
if line.startswith(key + "="):
|
|
return i, line.split("=", 1)[1]
|
|
return -1, None
|
|
|
|
ui, uval = value_of(UKEY)
|
|
pi, pval = value_of(PKEY)
|
|
if uval == user and pval is not None and matches(pval):
|
|
raise SystemExit(0) # converged - leave the file (and its salt) alone
|
|
|
|
salt = os.urandom(SALT_LEN)
|
|
digest = hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, DIGEST_LEN)
|
|
stored = base64.b64encode(salt).decode() + ":" + base64.b64encode(digest).decode()
|
|
desired = [UKEY + "=" + user, PKEY + '="@ByteArray(' + stored + ')"']
|
|
|
|
out = [line for i, line in enumerate(lines) if i not in (ui, pi)]
|
|
if SECTION in out:
|
|
at = out.index(SECTION) + 1
|
|
else:
|
|
if out and out[-1].strip():
|
|
out.append("")
|
|
out.append(SECTION)
|
|
at = len(out)
|
|
out[at:at] = desired
|
|
|
|
try:
|
|
os.makedirs(os.path.dirname(conf) or ".", exist_ok=True)
|
|
tmp = conf + ".webui-cred.new"
|
|
with open(tmp, "w", encoding="utf-8") as fh:
|
|
fh.write("\n".join(out) + "\n")
|
|
fh.flush()
|
|
os.fsync(fh.fileno())
|
|
# This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would
|
|
# cost the app the ability to save its own settings, so carry the ownership over.
|
|
try:
|
|
st = os.stat(conf)
|
|
os.chown(tmp, st.st_uid, st.st_gid)
|
|
os.chmod(tmp, stat.S_IMODE(st.st_mode))
|
|
except FileNotFoundError:
|
|
os.chown(tmp, int(os.environ.get("PUID", "1000")),
|
|
int(os.environ.get("PGID", "1000")))
|
|
os.chmod(tmp, 0o664)
|
|
os.replace(tmp, conf) # atomic: the app never reads a half-written conf
|
|
except OSError as exc:
|
|
sys.stderr.write("[qbt-webui-cred] cannot write %s: %s\n" % (conf, exc))
|
|
raise SystemExit(1)
|
|
raise SystemExit(10)
|
|
PY
|
|
rc=$?
|
|
|
|
if [ "$rc" = 0 ]; then
|
|
log "conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)"
|
|
elif [ "$rc" = 10 ]; then
|
|
log "asserted WebUI Username + Password_PBKDF2 into $CONF"
|
|
# postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY
|
|
# have read the old, credential-less conf and be serving a random temporary password.
|
|
# Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.
|
|
# Nothing to do when the service has not started yet - it will read the new file.
|
|
# Its own counter, NOT the shared `waited` API budget: sharing one counter is how the
|
|
# 2026-09-13 failure spent the entire verification budget before the first login and
|
|
# then fired attempts 2 and 3 with no wait left at all.
|
|
sv=0
|
|
while [ "$sv" -lt 60 ] && [ "$(s6-svstat -u "$SVC" 2>/dev/null)" != "true" ]; do
|
|
sv=$((sv + 1)); sleep 1
|
|
done
|
|
if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
|
|
log "qbittorrent-nox was already running - restarting it to load the credential"
|
|
s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)"
|
|
sleep 3
|
|
fi
|
|
else
|
|
log "FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep" >&2
|
|
log "minting a random password every start and BOTH arrs will break. Fix and restart." >&2
|
|
exit 0
|
|
fi
|
|
|
|
# (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL
|
|
# (shared ~120s budget across attempts), then spend exactly ONE login per attempt -
|
|
# at most 3 rejected logins, which stays under qBittorrent's default ban threshold.
|
|
#
|
|
# "THE API NEVER ANSWERED" IS A DIFFERENT FAULT FROM "THE PASSWORD WAS REJECTED", and
|
|
# conflating them is what made this hook lie once already (see the PORT comment). An
|
|
# unreachable WebUI is reported as unreachable, and the hook stops rather than spending
|
|
# its remaining login attempts to produce two more identical wrong diagnoses.
|
|
api_ok=0
|
|
while [ "$waited" -lt 120 ]; do
|
|
code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' "http://127.0.0.1:$PORT/" 2>/dev/null)
|
|
if [ -n "$code" ] && [ "$code" != "000" ]; then api_ok=1; break; fi
|
|
waited=$((waited + 1)); sleep 1
|
|
done
|
|
if [ "$api_ok" != 1 ]; then
|
|
log "the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the" >&2
|
|
log "credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was" >&2
|
|
log "never submitted. Do not rotate the credential on the strength of this line." >&2
|
|
exit 0
|
|
fi
|
|
body=$(curl -sS -m 10 \
|
|
--data-urlencode "username=$QBT_WEBUI_USERNAME" \
|
|
--data-urlencode "password=$QBT_WEBUI_PASSWORD" \
|
|
"http://127.0.0.1:$PORT/api/v2/auth/login" 2>/dev/null)
|
|
if [ "$body" = "Ok." ]; then
|
|
log "VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'"
|
|
exit 0
|
|
fi
|
|
log "WebUI rejected the durable credential (login said '$body')" >&2
|
|
# A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean
|
|
# the running qbittorrent-nox is serving something it read from an EARLIER conf, so
|
|
# rewriting the same bytes on the next attempt would change nothing. The remedy is to
|
|
# make the process RE-READ the file, so restart the supervised service and let the next
|
|
# attempt re-verify. Without this the retry loop could never change its own outcome.
|
|
if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
|
|
log "restarting qbittorrent-nox so it re-reads $CONF, then re-verifying" >&2
|
|
s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)" >&2
|
|
sleep 3
|
|
fi
|
|
done
|
|
|
|
log "GAVE UP after $attempt attempts: the WebUI does not accept the credential from git." >&2
|
|
# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container
|
|
# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.
|
|
# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the
|
|
# app, and the arrs' own download-client health check surfaces a broken credential.
|
|
exit 0
|
|
podSecurityContext:
|
|
# For the LONGHORN config PVC only, and for the same reason radarr/sonarr set it: fsGroup is a NO-OP
|
|
# on the NFS volume (the in-tree plugin does no ownership management), but on a Longhorn filesystem
|
|
# the kubelet applies it, so /config becomes group-writable by the now-uid-1000 process without
|
|
# depending on the image chowning it. (This image DOES chown: init-qbittorrent-config runs
|
|
# `lsiown -R abc:abc /config`. Belt and braces — the live /config was 911:911 before this change,
|
|
# and losing write access to it would lose the WebUI password hash and the session state.)
|
|
fsGroup: 3000
|
|
exposure: internal # LAN-only web UI at qbittorrent.${BASE_DOMAIN} (docs/13 row 13; R-DNS/R-NET)
|
|
egress: vpn # ALL public egress via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
|
|
storage:
|
|
- name: config
|
|
size: 5Gi
|
|
class: longhorn # settings/categories/session on 3-replica Longhorn; DR-critical (docs/04; R-STATE-02)
|
|
mountPath: /config # THE APP WRITES /config, NOT the renderer default /data/config.
|
|
# LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
|
|
# /config), so without this the Longhorn PVC mounted at /data/config sat
|
|
# holding nothing but lost+found while the app wrote its entire
|
|
# configuration to the container's EPHEMERAL writable layer — destroyed on
|
|
# every pod recreation: restart, reschedule, image bump. Measured in-cluster
|
|
# 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
|
|
# dev=2053 on the node's own disk with [Sentry asp config.xml logs].
|
|
# This is the SAME failure catalog/plex.yaml:54 already records fixing.
|
|
# The live /config contents were copied onto each PVC before this landed,
|
|
# so API keys and credentials other apps reference were preserved.
|
|
# THE DOWNLOADS AREA — ONE mount of the SAME export sonarr/radarr/plex already consume
|
|
# (192.168.2.102:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
|
|
# /downloads, which is precisely where this app is ALREADY configured to write.
|
|
#
|
|
# WHY subPath: downloads AND NOT THE EXPORT ROOT AT /media (the shape sonarr/radarr use):
|
|
# * LEAST PRIVILEGE, AND IT COSTS NOTHING HERE. A download client has no business writing the
|
|
# LIBRARY. The arrs need the root mount because THEY hardlink downloads/ -> tv/ or movies/ and
|
|
# link() is a same-MOUNT operation (EXDEV otherwise) — their smallest common ancestor genuinely
|
|
# IS the export root. qBittorrent performs no such cross-directory link: it writes into the
|
|
# downloads tree and nothing else, so narrowing to that subtree removes movies/, tv/ and photos/
|
|
# from this pod's mount namespace without removing any operation it actually performs.
|
|
# * HARDLINKS STILL WORK, because they are not done here. The arrs still see ONE mount containing
|
|
# both /media/downloads and /media/tv, so their link() stays intra-mount. Proven live 2026-09-12:
|
|
# a file created from this pod under /downloads was hardlinked from the sonarr pod between
|
|
# /media/downloads/... and /media/tv/... with st_nlink 1 -> 2 and no EXDEV.
|
|
# * IT IS ONE MOUNT, so the L0 test `bulk-tier-distinct-from-replicated` assertion (6) — no
|
|
# workload mounts one export twice — holds. That assertion is about TWO mounts inside ONE pod;
|
|
# a single subPath mount is not the EXDEV trap, two are.
|
|
# * The narrowing is a CLIENT/kubelet guarantee, stated honestly: the NFS server still exports the
|
|
# whole dataset to this node, so the containment is the mount namespace, not the server's ACLs —
|
|
# the same class of caveat docs/BULK-NFS-STORAGE.md §5 records for plex's readOnly.
|
|
#
|
|
# NOT readOnly — this is the one workload whose whole job is WRITING here.
|
|
#
|
|
# KNOWN, DELIBERATE SIDE EFFECT ON SHARED STATE: the image's init-qbittorrent-config runs
|
|
# `lsiown abc:abc /downloads` (non-recursive) whenever /downloads is a mount, so starting this pod
|
|
# chgrps /mnt/Storage/plex/downloads from 1000:1000 to 1000:3000 on the SERVER. Benign and verified:
|
|
# the directory is mode 0755, whose group bits (r-x) are identical to its other bits, so no consumer
|
|
# — k3s or legacy — gains or loses any access. Recorded because it is a write to state shared with
|
|
# non-cluster consumers, which should never be a surprise.
|
|
#
|
|
# PATH AGREEMENT WITH THE ARRS IS STILL *NOT* CLOSED — see DECISIONS-LOG 2026-09-12 "Remote Path
|
|
# Mappings". This pod calls the finished file /downloads/<name>; sonarr/radarr call the same bytes
|
|
# /media/downloads/<name>. Same inode, different absolute path, so Completed Download Handling needs
|
|
# ONE Remote Path Mapping per arr (remote host `qbittorrent.qbittorrent.svc.cluster.local`, remote
|
|
# path /downloads/, local path /media/downloads/).
|
|
#
|
|
# THOSE MAPPINGS ARE NOW CONFIGURED AND VERIFIED ON PROD (2026-09-12), together with the qBittorrent
|
|
# download client itself — which did not exist in EITHER arr until then (both
|
|
# `GET /api/v3/downloadclient` returned `[]`, so completed-download handling had never been wired up
|
|
# at all). `downloadclient/testall` reports isValid=true in both, and Sonarr's queue was observed
|
|
# reporting outputPath=/downloads/... with the mapping deleted and /media/downloads/... with it
|
|
# present — same torrent, same client.
|
|
#
|
|
# BUT THEY LIVE IN THE ARRS' OWN DATABASES (app state on their config PVCs), NOT IN GIT: a rebuild
|
|
# from git alone does NOT restore them and no gate here notices their absence. The alternative —
|
|
# repointing qBittorrent's SavePath at /media/downloads and mounting this subPath there instead —
|
|
# buys path agreement at the cost of editing THIS app's config PVC, which is also app state; it was
|
|
# not chosen because it changes a running download client's save path, which the deferral above
|
|
# already shows is easy to get wrong. It is nonetheless the RECOMMENDED end state (it makes the
|
|
# mappings unnecessary and puts the mountPath half of the fix in git) and is awaiting an operator
|
|
# decision, not a technical blocker.
|
|
#
|
|
# RELATED OPEN BLOCKER: this app has no durable WebUI credential (no WebUI\Username /
|
|
# WebUI\Password_PBKDF2 in qBittorrent.conf), so v5 issues a NEW temporary `admin` password on every
|
|
# start. The arrs' saved download-client password therefore goes stale on every pod restart. See the
|
|
# same DECISIONS-LOG entry for the two fixes, neither of which was taken unilaterally.
|
|
- name: downloads
|
|
class: bulk
|
|
export: /mnt/Storage/plex # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER} — the SAME export
|
|
# plex (ro) and sonarr/radarr (rw) mount; one dataset, one set of inodes
|
|
subPath: downloads # the handoff subtree ONLY: no movies/, no tv/, no photos/
|
|
mountPath: /downloads # matches the app's EXISTING Session\DefaultSavePath / TempPath, so no
|
|
# change to config-PVC app state is needed to make this take effect
|
|
size: 9Ti # ADVISORY ONLY on a static NFS PV (the ZFS pool is the real limit): the
|
|
# dataset's 8.4 T used rounded up, stated the same way plex/sonarr/radarr
|
|
# state it so `kubectl get pv` reports something true
|
|
dependencies: [] # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02).
|
|
# STILL EMPTY WITH THE NFS MOUNT ABOVE, AND THAT IS CORRECT, NOT AN
|
|
# OVERSIGHT. An NFS PersistentVolume is mounted by the NODE KERNEL in the
|
|
# HOST netns, not by this pod: there is no pod-side socket to 2049, so no
|
|
# CiliumNetworkPolicy can permit or deny it and none is needed. Verified by
|
|
# construction — sonarr and radarr have mounted this same export read-write
|
|
# for weeks and neither has any CNP naming the NFS server or port 2049
|
|
# (grep apps/wave-1/sonarr.yaml: none). The SAME fact is what keeps the VPN
|
|
# egress gateway off this traffic (its selector is a podSelector), and
|
|
# 192.168.2.102 is inside the 192.168.0.0/16 excludedCIDR besides — see
|
|
# platform/vpn-egress/egress-steering.yaml, "BULK NFS".
|
|
ingressRoutes:
|
|
- host: qbittorrent.${BASE_DOMAIN}
|
|
port: 8080 # qBittorrent WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
|
|
auth:
|
|
mode: forward-auth # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)
|
|
groups:
|
|
- media-users # authenticated AND in an authorized group (R-IDP-06)
|
|
# See catalog/jackett.yaml for the full rationale: --allowed-group is repeatable and OR-ed, so
|
|
# platform-admins here is a superset, not a widening (that group already carries cluster-admin
|
|
# via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
|
|
# immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
|
|
- platform-admins # operators of these services must not be locked out of them
|
|
# NO `apiRoutes` HERE: UNMEASURED, not cleared. qBittorrent's WebUI drives everything through
|
|
# /api/v2/, so it is plausibly affected in the same way jackett is — but plausibility is what
|
|
# put a renderer constant on the table in the first place. MEASURED 2026-10-01 on the live pod
|
|
# (qbittorrent-5547bdb9bd-zq557, 0 restarts, 23h): 13,649 log lines, ZERO unauthenticated
|
|
# requests, nothing but health probes. No traffic means no evidence either way. Add `^/api/`
|
|
# when someone opens the WebUI and records the 302/401 split, as was done for jackett.
|
|
update:
|
|
channel: stable # R-UPD-01
|
|
range: ">=5.0.0 <6.0.0" # stay on the qBittorrent 5.x stable line
|
|
cadence: "0 4 * * *"
|
|
autoRollbackOnFailedHealth: true # R-UPD-03
|
|
observability:
|
|
metrics: true
|
|
slis:
|
|
- up # download client down -> alert (docs/10; R-OBS-01)
|
|
backup:
|
|
policy: daily-offcluster # config (categories/settings) backed up off-cluster (docs/04; R-STATE-03)
|