plex, sonarr, radarr and nextcloud each gain one bulk volume. Authored in the catalog and re-rendered, never hand-edited into apps/ (render-golden: 10/10). Layout, the export change the operator must apply, the uid analysis and the validation probes are in docs/BULK-NFS-STORAGE.md; everything there was measured on the host over read-only SSH, not assumed. WHAT THE LAYOUT TURNED OUT TO BE, AND WHY IT DECIDED THE DESIGN /mnt/Storage/plex holds movies/ tv/ photos/ AND downloads/ as SIBLINGS in one dataset — so the arrs' import source and their libraries are one filesystem on the server, and the download client's completed folders are already named radarr/ and tv-sonarr/. The k3s arrs are joining a tree an arr stack has been driving for a long time. That sibling fact is the whole argument for the arrs mounting the export ROOT rather than a subPath per directory. Sonarr and Radarr import by hardlinking or atomically renaming, both same-filesystem operations, and to the kernel "same filesystem" means SAME MOUNT. Two subPath mounts of one export are two mounts: link() across them returns EXDEV regardless of what the server knows, and the arrs do not fail loudly — they degrade to copy-then-delete, rewriting every file over NFS and destroying the seeding copy. The smallest subtree containing both tv/ and downloads/ IS the export root, so the root mount is the MINIMAL mount that preserves hardlinks. Least privilege bounded below by a hard requirement. The cost is stated, not hidden: sonarr can also write /media/movies. plex /mnt/Storage/plex -> /media ro ReadOnlyMany sonarr /mnt/Storage/plex -> /media rw ReadWriteMany radarr /mnt/Storage/plex -> /media rw ReadWriteMany nextcloud /mnt/Storage/nextcloud -> /mnt/legacy-nextcloud ro ReadOnlyMany (subPath: nextcloud) Plex is read-only because it indexes and streams; every write on that tree belongs to the arrs, and saying so removes Plex from the set of things that can damage 8.33 T. All three media consumers use the SAME mountPath deliberately: when qbittorrent is eventually given this export, the arrs and the download client will already agree on absolute paths, which is the other precondition for hardlinked imports. *** /media AND NOT /data — A COLLISION CAUGHT WHILE RENDERING. *** The arrs' Longhorn config PVC renders at the renderer default /data/config. A media mount at /data is mounted OVER it and MASKS the config volume: the pod comes up with its app database shadowed by an 8 T media tree. That discovery also exposed a SEPARATE PRE-EXISTING BUG, reported and deliberately NOT fixed here: LinuxServer arr images run with -data=/config, so /data/config is very likely the wrong home for that PVC and the app has been writing its EPHEMERAL layer — the exact failure catalog/plex.yaml's mountPath comment records already having been bitten by. Plex was fixed; sonarr, radarr, qbittorrent and jackett were not. Fixing it starts those apps from an empty config volume, so it needs its own change with the operator present. This change does not make it worse. UID/GID — THE FAILURE THIS TIER ALWAYS HAS The export is sec=sys + no_root_squash, so the client's numeric uid passes through verbatim and the server's ownership is the entire access story. The tree is 0755 owned 1000:3000, so ONLY uid 1000 can write it. LinuxServer images default to 911. Without intervention sonarr and radarr would have mounted cleanly, gone Ready, and failed EVERY import with EACCES. And fsGroup cannot fix that — the in-tree NFS plugin does NO ownership management, so podSecurityContext.fsGroup is a no-op on an NFS volume. Reaching for it is how this gets misdiagnosed. So PUID=1000/PGID=3000 are added to both arrs (PGID 3000 keeps what they write group-readable to the live non-k3s consumers of the same export), and fsGroup: 3000 is added for the LONGHORN config PVC only, so changing PUID does not depend on the image re-chowning /config. plex already runs as 1000 and reads a 0755 tree; nextcloud reads as www-data. Neither needs work. NEXTCLOUD IS MOUNTED READ-ONLY, WHICH IS A DEVIATION FROM THE BRIEF — ON EVIDENCE The task said nextcloud writes. The dataset says otherwise: of 270 G, 267 G is config_backup/, the rest is config/ (103 M) and config_old/ (3.4 G). There is NO data/ directory. This is the OLD non-k3s instance's configuration plus a backup blob — an archive — and that instance is STILL LIVE on it (exported rw to 192.168.1.6 and .5, which the operator has said must stay active). Two Nextclouds writing one instance's config directory, each with its own database and therefore its own oc_filecache, is unrecoverable corruption. "Do not grant more than needed" lands on ro, and ro is the trivially reversible choice. If the intent is in fact a MIGRATION, that is a larger and more deliberate change than a mount; the one-line flip is documented. Read-only for nextcloud can additionally be enforced SERVER-side (only nextcloud consumes that dataset from k3s), and the sharenfs value in the doc uses ro= for the k3s clients. The media tree cannot: plex and the arrs may land on any node, so the server sees the same addresses for reader and writers. Plex's ro is therefore a client-side guarantee — stated as a limitation, not left implied. Co-Authored-By: Claude Opus 5 <[email protected]>
110 lines
7.6 KiB
YAML
110 lines
7.6 KiB
YAML
# ServiceDefinition: plex
|
|
# Source of truth: spec docs/13-service-catalog.md, row 1 (the catalog's largest service).
|
|
# public | egress internet | config on Longhorn (metadata DB, DR-noncritical/reacquirable) |
|
|
# auth: native (Plex accounts, NOT Keycloak — docs/13 + docs/15 documented exception).
|
|
# Validates against schemas/servicedefinition.schema.json.
|
|
#
|
|
# BULK MEDIA — NOW ATTACHED (docs/04 bulk tier; the long-standing OQ-2 prerequisite, resolved for
|
|
# the STATIC case). The library is 8.33 T of ZFS on the Proxmox host at /mnt/Storage/plex, and it
|
|
# already exists: nothing is provisioned, an existing dataset is mounted. See
|
|
# docs/BULK-NFS-STORAGE.md for the layout, the export change and the uid analysis. (The bulk-tier
|
|
# requirement ID is deliberately still not typed here — see that document's "gate" note.)
|
|
name: plex
|
|
kind: container
|
|
source:
|
|
image: plexinc/pms-docker:latest
|
|
# plexinc/pms-docker env. Rendered VERBATIM onto the app container by render.py (schema env list).
|
|
env:
|
|
# PLEX_UID/PLEX_GID: the image boots as root (s6 PID1), chowns /config to this uid/gid, then drops
|
|
# the "Plex Media Server" process to it. 1000 matches the Longhorn ownership convention the other
|
|
# stateful apps use (seerr/semaphore run as UID/GID 1000), keeping the /config PVC writable by the
|
|
# dropped plex user. (Plex has no podSecurityContext/fsGroup — the image manages ownership itself.)
|
|
- name: PLEX_UID
|
|
value: "1000"
|
|
- name: PLEX_GID
|
|
value: "1000"
|
|
# TZ: no other service in the repo sets a timezone (repo-wide grep: none), so use the safe default.
|
|
- name: TZ
|
|
value: Etc/UTC
|
|
# ALLOWED_NETWORKS marks these CIDRs as trusted LAN, so in-cluster clients (Seerr -> plex:32400) are
|
|
# treated as local and NOT pushed onto Plex's remote-auth/relay path. 10.0.0.0/8 covers the Cilium
|
|
# pod CIDR (10.0.0.0/16, CLUSTER_POD_CIDR — Seerr's pod is 10.0.x.x) and the k3s service net
|
|
# (10.43.0.0/16); 192.168.0.0/16 covers the home LAN.
|
|
- name: ALLOWED_NETWORKS
|
|
value: 10.0.0.0/8,192.168.0.0/16
|
|
# PLEX_CLAIM is deliberately NOT set here: a https://plex.tv/claim token has a ~4-min TTL, far
|
|
# shorter than the commit -> push -> Flux reconcile -> pod-roll cycle, so any token committed to git
|
|
# is already expired by the time the pod starts. Claiming is a one-time LIVE step (UI sign-in, or a
|
|
# single API POST) against the running server; the claim then persists on the /config PVC and
|
|
# survives pod rolls (including the roll this env change causes), so it never belongs in git.
|
|
# ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex
|
|
# which URL to advertise to plex.tv for REMOTE clients. The only candidate is
|
|
# https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL
|
|
# Gateway VIP (conformance/tests/l2-behavioral/dns-internal-resolution), and both its public
|
|
# reachability and whether the TLS-terminating Cilium Gateway proxies Plex's 32400 API correctly for
|
|
# remote access are UNVERIFIED. Advertising a wrong/internal-only URL breaks remote access for ALL
|
|
# clients, so per "prefer safety over completeness" it is left unset. FOLLOW-UP: once remote
|
|
# reachability through the Gateway is confirmed, set ADVERTISE_IP: "https://plex.home.keithjackson.dev:443".
|
|
exposure: public # edge port-forward + public DNS; clients + Seerr reach it (docs/09; R-DNS/R-NET)
|
|
egress: internet # normal WAN egress (docs/03; R-NET-05)
|
|
storage:
|
|
- name: config
|
|
size: 8Gi
|
|
class: longhorn # Plex DB/metadata: replicated app state on the 3-replica tier (docs/04; R-STATE-01/02)
|
|
mountPath: /config # plexinc/pms-docker writes its identity/claim + "Library" DB under /config, NOT the
|
|
# renderer default /data/config. Without this the PVC sat unused and Plex wrote to the
|
|
# container's EPHEMERAL layer -> unclaimed + all state lost on restart. The image inits as
|
|
# root (s6-svscan PID 1), chowns /config to the plex user, then drops -> no fsGroup needed
|
|
# (verified in-cluster: PID1=root, "Plex Media Server"=plex). Persisting /config here also
|
|
# makes the claim survive restarts once claimed. To auto-claim on first boot instead of the
|
|
# UI sign-in, the operator adds a PLEX_CLAIM env from a fresh https://plex.tv/claim token
|
|
# (4-min TTL) — deliberately NOT wired here: the env secretKeyRef pattern has no `optional`,
|
|
# so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
|
|
# THE LIBRARY (docs/04 bulk tier). One READ-ONLY mount of the export ROOT, giving /media/movies,
|
|
# /media/tv and /media/photos — the whole tree the layout actually has (`ls /mnt/Storage/plex`:
|
|
# downloads/ movies/ photos/ tv/).
|
|
#
|
|
# WHY THE ROOT AND NOT THREE subPath MOUNTS. Nothing is gained by narrowing: the mount is `ro`,
|
|
# so no subPath could remove a privilege that matters, while three mounts would triple the NFS
|
|
# mounts per pod and mean a manifest change every time a top-level library is added.
|
|
#
|
|
# WHY READ-ONLY, when the export is `rw` and the other consumers write. Plex is a SERVER: it
|
|
# indexes and streams. Every write on that tree belongs to Sonarr and Radarr. Marking it here
|
|
# costs nothing and removes Plex from the set of things that can damage 8.33 T. Enforced by the
|
|
# PV's nfs.readOnly (a real `ro` mount), not only by the volumeMount flag.
|
|
#
|
|
# UID: no change needed. The tree is 0755 owned 1000:3000 throughout, so it is world-readable and
|
|
# world-traversable, and PLEX_UID/PLEX_GID above already make the process uid 1000 in any case.
|
|
# (fsGroup would have been a no-op here regardless — the in-tree NFS plugin does no ownership
|
|
# management. That is the trap the arrs' PUID/PGID exists to avoid; Plex never hits it.)
|
|
- name: media
|
|
class: bulk
|
|
export: /mnt/Storage/plex # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER}
|
|
mountPath: /media
|
|
readOnly: true
|
|
size: 9Ti # ADVISORY ONLY (nothing enforces a static NFS PV's capacity): the
|
|
# dataset's 8.33 T used, rounded up, so `kubectl get pv` reports
|
|
# something true. The real limit is the ZFS pool.
|
|
dependencies: [] # default-deny: reaches only DNS; clients/Seerr reach IT (docs/03; R-NET-01/02)
|
|
ingressRoutes:
|
|
- host: plex.${BASE_DOMAIN}
|
|
port: 32400 # Plex web UI + API served on 32400 (wildcard TLS at the Gateway; docs/08/09)
|
|
auth:
|
|
mode: native # Plex account auth — documented exception, NOT Keycloak (docs/13/15; R-IDP-02)
|
|
update:
|
|
channel: stable # R-UPD-01 (never pre-release)
|
|
range: ">=1.40.0" # acceptable versions (advisory; image-automation off during onboarding)
|
|
cadence: "0 4 * * 0" # weekly stable check/apply
|
|
autoRollbackOnFailedHealth: true # R-UPD-03
|
|
observability:
|
|
metrics: true
|
|
slis:
|
|
- up # R-OBS-01: service-down alert
|
|
backup:
|
|
# The config volume is on the longhorn (replicated, "irreplaceable") tier, so the L0 Kyverno
|
|
# backup policy (R-STATE-03) forbids `reacquirable` here — it must reference an off-cluster
|
|
# schedule. Plex's metadata DB (library index, thumbnails, watch state) is expensive to rebuild,
|
|
# so it is backed up off-cluster. The bulk library itself, once attached, is the `reacquirable`
|
|
# part (docs/04) and is NOT covered by this schedule.
|
|
policy: daily-offcluster # off-cluster backup of the longhorn config volume (docs/04; R-STATE-03)
|