87c3d83dff827c61ff65bc667900a95509ac10e4
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
2f38b8156b |
fix(arr): wire qBittorrent as a download client + Remote Path Mappings
qBittorrent reports finished releases at /downloads/<name>; sonarr and radarr
see the same inodes at /media/downloads/<name>, so Completed Download Handling
could never resolve what the client handed it. Configured, on prod, via each
app's own HTTP API:
* download client qBittorrent -> qbittorrent.qbittorrent.svc.cluster.local:8080
(it did not exist in EITHER arr: both /api/v3/downloadclient returned [])
* Remote Path Mapping /downloads/ -> /media/downloads/ for that host
Both built from each app's /downloadclient/schema, matched on identity before
writing, and re-running creates nothing. downloadclient/testall reports
isValid=true in both. Proven with a negative control: Sonarr's queue showed
outputPath=/downloads/... with the mapping deleted and /media/downloads/...
with it present, for the same completed torrent.
This state lives in sonarr.db / radarr.db on the config PVCs, NOT in git — a
rebuild from git alone does not restore it. Recorded as a wart, with path
agreement (repointing SavePath to /media/downloads) recommended as the durable
fix and put to the operator rather than taken unilaterally.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
78bc53afc7 |
feat(qbittorrent): attach the downloads area of the bulk export — the chain was broken end to end
qBittorrent was the one member of the download chain with NO media volume.
Its live config said
Session\DefaultSavePath=/downloads/
Session\TempPath=/downloads/incomplete/
Downloads\SavePath=/downloads/
while `ls /downloads` in the container returned "No such file or directory".
The catalog entry declared exactly one volume — config at /config on longhorn —
and carried a comment deferring the downloads volume "until OQ-2 is resolved".
OQ-2 WAS resolved (schema 1.5.0 static bulk NFS PVs); plex, sonarr, radarr and
nextcloud all took their mounts and qBittorrent's was simply never added. So the
download client was writing, or failing to write, a path that did not exist —
and anything it did create would have been invisible to sonarr/radarr, lost on
every pod recreation, and consuming the node's ephemeral disk until then.
WHAT THIS ADDS to catalog/qbittorrent.yaml (rendered into apps/wave-0):
* storage[downloads]: class bulk, export /mnt/Storage/plex, subPath downloads,
mountPath /downloads, 9Ti advisory. ONE mount of the SAME export sonarr,
radarr (rw) and plex (ro) already consume, narrowed to the handoff subtree.
* env PUID=1000 / PGID=3000. LOAD-BEARING, not cosmetic: the container ran as
uid 911 and /mnt/Storage/plex/downloads is 0755 owned 1000:1000, so the
export would have mounted cleanly and then failed every write with EACCES.
Measured both ways on prod: a write as uid 1000 succeeded, the same write as
uid 911 was denied.
* podSecurityContext.fsGroup 3000, for the LONGHORN config PVC only, exactly as
sonarr/radarr set it — fsGroup is a no-op on an in-tree NFS volume.
WHY subPath AND NOT THE EXPORT ROOT AT /media (sonarr/radarr's shape): a download
client has no business writing the library. The arrs need the root mount because
THEY hardlink downloads/ -> tv/|movies/ and link() is a same-MOUNT operation;
qBittorrent performs no such link, so narrowing to downloads/ removes movies/,
tv/ and photos/ from its mount namespace without removing any operation it
performs. It is still ONE mount of the export, so the L0 assertion that no
workload mounts one export twice (the EXDEV trap) holds — and the arrs' own
link() stays intra-mount. Proven live: nlink 1 -> 2, same inode, no EXDEV, every
step executed as uid 1000.
NETWORK: nothing to grant and nothing to steer. An NFS PersistentVolume is
mounted by the NODE KERNEL in the host netns — the established socket to
192.168.2.102:2049 is on the node's own LAN address, the pod netns has no socket
to 2049 at all, and no CiliumNetworkPolicy on this cluster names that host or
port. `ip route get 192.168.2.102` picks eno1 even on the tunnel-owning node, and
Cilium's own egress map lists 192.168.0.0/16 as an Excluded CIDR for every
vpn-class pod while 0.0.0.0/0 goes to the gateway. Public egress still exits the
PIA address; the VPN guarantee is untouched.
NOT CLOSED BY THIS COMMIT, deliberately: absolute paths still differ between this
pod (/downloads/x) and the arrs (/media/downloads/x), so Completed Download
Handling needs one Remote Path Mapping per arr. That lives in the arrs' own
databases, not in git.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
5da116b34d |
fix(oauth2-proxy): platform-admins may reach the media UIs — the immediate 403
The four forward-auth media UIs (jackett, qbittorrent, radarr, sonarr) gate on `--allowed-group=media-users` alone. On prod the `homelab` realm has exactly one user — the operator's `jakman85` — and it is a member of `platform-admins` ONLY. `media-users` has zero members, so those four hosts were reachable by nobody: oauth2-proxy completed OIDC discovery, redirected to Keycloak, got a token whose `groups` claim is `["platform-admins"]`, found no allowed group in it, and answered 403. No login form appeared because the browser already held a Keycloak SSO session from a platform-admins login — hence "403 immediately". pihole and crafty were unaffected: they already gate on platform-admins. `--allowed-group` is repeatable and OR-ed, so listing both groups means "media consumers OR platform operators". This grants no real privilege — platform-admins already maps to cluster-admin via k8s-oidc-rbac.yaml (oidc:platform-admins), so a member can already exec into these pods — and it keeps media-users meaningful for non-admin household accounts. Ruled out while diagnosing: client-secret drift (all 7 sidecar secrets hash-match keycloak-realm-clients), stale apex URLs (every live sidecar carries keycloak.prod.home.keithjackson.dev), missing group mappers (all 7 clients emit claim=groups, full.path=false), and the central oauth2-proxy pod restarts — those were Keycloak returning 503 "no healthy upstream" to OIDC discovery at startup, a consequence of the Keycloak roll, not a cause. Goldens re-rendered with the canonical command: python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation R-IDP-06 Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
6b953cc56b |
fix(arrs): the config PVC was mounted where the app never looks
sonarr, radarr, jackett and qbittorrent all mounted their Longhorn config PVC at the renderer's default /data/config, but every one of these images runs with `-data=/config`. So the PVC held nothing but lost+found while the app wrote its entire configuration to the container's EPHEMERAL writable layer — destroyed on every pod recreation: restart, reschedule, image bump. Measured in-cluster before fixing, per app, rather than inferred: /data/config dev=2080 [lost+found] <- the Longhorn PVC /config dev=2053 [Sentry asp config.xml logs ...] <- the node's own disk and a marker file written to /config did not appear in /data/config, so they are demonstrably different storage. qbittorrent's /config sat on dev=66309 (/dev/nvme0n1p5), i.e. whichever node it happened to land on. This is the SAME failure catalog/plex.yaml:54 already records having fixed for Plex; plex was the only one of the five that was right. The live /config contents were copied onto each PVC BEFORE this landed, so jackett's API key and qbittorrent's credentials — which other apps reference — survive the move rather than being regenerated. catalog/sonarr.yaml:61 documented this as a latent bug and deliberately left it because it changes live on-disk state. It is being fixed now because all four pods had just been recreated, so their configs were freshly generated and nothing had accumulated to lose. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
c8c1d2935d |
feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain
INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).
THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.
Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.
PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.
BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.
TWO THAT CANNOT BE TOKENIZED, flagged not forced:
- headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
anyway — client secrets are realm-scoped.
- kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
prod.home.
EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.
I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.
DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.
L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.
PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.
Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
8392b52ee8 |
Onboard VPN downloaders (qBittorrent + Jackett) via ServiceDefinition renderer
Add catalog/ ServiceDefinitions for qbittorrent and jackett and their rendered apps/ manifests, wired into apps/kustomization.yaml. Both are VPN-class (egress: vpn -> egress-class=vpn pod label steered by the CiliumEgressGatewayPolicy + host kill-switch, ADR-0015) and internal forward-auth UIs (oauth2-proxy -> Keycloak, group media-users). Config on 3-replica Longhorn. qBittorrent's bulk downloads volume is DEFERRED pending OQ-2 (non-provisioning bulk StorageClass stub), mirroring plex/arr. Traces: R-NET-05/06 (VPN egress + kill-switch), R-IDP-02/06 (forward-auth), R-STATE-02/03, R-OBS-01, R-UPD-01/03, R-GITOPS-02. Co-Authored-By: Claude Opus 4.8 <[email protected]> |