home-network-k3s-impl/catalog/plex.yaml

# ServiceDefinition: plex
# Source of truth: spec docs/13-service-catalog.md, row 1 (the catalog's largest service).
# public | egress internet | config on Longhorn (metadata DB, DR-noncritical/reacquirable) |
# auth: native (Plex accounts, NOT Keycloak — docs/13 + docs/15 documented exception).
# Validates against schemas/servicedefinition.schema.json.
#
# BULK MEDIA — NOW ATTACHED (docs/04 bulk tier; the long-standing OQ-2 prerequisite, resolved for
# the STATIC case). The library is 8.33 T of ZFS on the Proxmox host at /mnt/Storage/plex, and it
# already exists: nothing is provisioned, an existing dataset is mounted. See
# docs/BULK-NFS-STORAGE.md for the layout, the export change and the uid analysis. (The bulk-tier
# requirement ID is deliberately still not typed here — see that document's "gate" note.)
name: plex
kind: container
source:
  image: plexinc/pms-docker:latest
# plexinc/pms-docker env. Rendered VERBATIM onto the app container by render.py (schema env list).
env:
  # PLEX_UID/PLEX_GID: the image boots as root (s6 PID1), chowns /config to this uid/gid, then drops
  # the "Plex Media Server" process to it. 1000 matches the Longhorn ownership convention the other
  # stateful apps use (seerr/semaphore run as UID/GID 1000), keeping the /config PVC writable by the
  # dropped plex user. (Plex has no podSecurityContext/fsGroup — the image manages ownership itself.)
  - name: PLEX_UID
    value: "1000"
  - name: PLEX_GID
    value: "1000"
  # TZ: no other service in the repo sets a timezone (repo-wide grep: none), so use the safe default.
  - name: TZ
    value: Etc/UTC
  # ALLOWED_NETWORKS marks these CIDRs as trusted LAN, so in-cluster clients (Seerr -> plex:32400) are
  # treated as local and NOT pushed onto Plex's remote-auth/relay path. 10.0.0.0/8 covers the Cilium
  # pod CIDR (10.0.0.0/16, CLUSTER_POD_CIDR — Seerr's pod is 10.0.x.x) and the k3s service net
  # (10.43.0.0/16); 192.168.0.0/16 covers the home LAN.
  - name: ALLOWED_NETWORKS
    value: 10.0.0.0/8,192.168.0.0/16
  # PLEX_CLAIM — the ONE-SHOT plex.tv server claim, sourced from the SOPS Secret `plex-claim`
  # (apps/wave-0/plex-claim-secret.sops.yaml, same wave => applied in the same reconcile as this
  # workload). Plex cannot create libraries over the mounted bulk share while the server is
  # UNCLAIMED, so this is wired; it was previously left out on the reasoning below, and only one
  # half of that reasoning survived contact with the upstream source.
  #
  # A SHORT TTL IS THE OPERATIONAL CONSTRAINT, not a reason to abstain. A https://plex.tv/claim token
  # expires in MINUTES — read the countdown the claim page renders; do not budget against a number
  # written in this tree. (This comment previously asserted "~4 minutes" as established fact. It was
  # UNCITED, upstream plexinc/pms-docker documents no expiry at all, and that number was load-bearing
  # for the whole procedure design — including a since-disproven claim that the delivery pipeline was
  # too slow to fit inside it. Measured 2026-09-29: the full loop is ~2m45s and fits.)
  #
  # The live token is injected by the operator inside that window, with an explicit `flux reconcile`
  # to collapse the wait. Scripted end to end as infra/plex/claim-ship.sh; runbooks/plex-claim.md is
  # the procedure. SERVER CLAIMED 2026-09-29 — this env is now a permanent no-op (see below), and the
  # committed value is the spent token from that claim rather than the original placeholder.
  #
  # ONE-SHOT, AND SAFE TO SIT HERE EXPIRED. Verified from plexinc/pms-docker's own first-run script
  # (branch `master`, root/etc/cont-init.d/40-plex-first-run, confirmed byte-identical to the file in
  # the running `:latest` image), NOT assumed:
  #   - the exchange (line 106) is guarded on TWO conditions: PLEX_CLAIM non-empty AND the
  #     PlexOnlineToken pref in /config empty. Once claimed, the block is SKIPPED — this env is a
  #     no-op on every subsequent start, whatever it holds. (Upstream README: "If server is already
  #     logged in, this parameter is ignored.") The claim itself persists on the /config PVC as before.
  #   - an invalid/expired token on a still-unclaimed /config gets a 403 with an empty body; curl's
  #     status is unchecked, the token `sed` yields "", the success branch at line 120 does not fire,
  #     there is no `else` and no `exit 1`, and the script (no `set -e`) exits 0. It comes up UNCLAIMED and
  #     SERVING — it does NOT crashloop. The image also leaves S6_BEHAVIOUR_IF_STAGE2_FAILS at its
  #     default 0, which continues startup even when a cont-init script fails.
  # So the earlier "a placeholder claim Secret would crashloop the pod" note was WRONG about the
  # placeholder; it was right only about an ABSENT Secret. secretKeyRef here has no `optional` field,
  # so the Secret must always exist in the Flux graph — which is why it is committed, encrypted, and
  # listed in apps/wave-0/kustomization.yaml rather than hand-created at claim time. No schema change
  # is needed: `optional` would only matter if a missing Secret had to be tolerated, and it doesn't.
  - name: PLEX_CLAIM
    valueFrom:
      secretKeyRef:
        name: plex-claim
        key: PLEX_CLAIM
  # ADVERTISE_IP ("Custom server access URLs") — NOW SET. This replaces a DEFERRAL note that stood
  # here until 2026-10-04, and the deferral's premise has changed: it weighed only ONE candidate,
  # `https://plex.home.keithjackson.dev` through the L7 Gateway. The operator chose the NATIVE L4
  # path instead on 2026-10-03 (apps/wave-0/plex-remote.yaml) so streams go straight to
  # ${PLEX_L4_VIP}:32400 and bypass the Envoy hop. What was still missing was a URL whose TLS
  # CERTIFICATE validates — and getting that wrong is what took the service down.
  #
  # THE VALUE IS A `plex.direct` NAME, NOT THE RAW VIP, AND THAT IS THE WHOLE POINT. Plex serves
  # 32400 with its own certificate; measured live 2026-10-04 it carries exactly one name:
  #     CN = *.<serverhash>.plex.direct      (one SAN, no IP SANs at all)
  # so `https://<ip>:32400` can NEVER validate. `<dashed-ip>.<serverhash>.plex.direct` is Plex's own
  # mechanism for this: it is PUBLIC DNS that resolves to the embedded address, so the name points at
  # the LAN VIP while the wildcard cert above matches it. Measured 2026-10-04 with FULL verification
  # (no `curl -k`):
  #     https://192-168-2-24.<hash>.plex.direct:32400/identity -> 200, TLS verify OK
  #     https://192.168.2.24:32400/identity                    -> 000, cert validation FAILED
  #
  # THIS FIELD EXISTS BECAUSE AUTO-DISCOVERY CANNOT WORK HERE. Left empty, Plex publishes the
  # addresses it can see itself — which from a pod is the overlay IP (`10.0.2.136`), useless to a LAN
  # client. The VIP has to be advertised explicitly; there is no "just clear it" fix.
  #
  # WHAT WENT WRONG, recorded so it is not re-introduced by hand. customConnections OVERRIDES Plex's
  # own publishing, and on 2026-10-04 it held two hand-typed URLs, both uncompletable:
  #   - `https://192.168.2.24:32400`              -> cert cannot cover an IP. Measured: a storm of
  #     `CERT: incomplete TLS handshake ... sslv3 alert certificate unknown` in Plex's log every few
  #     seconds, from REAL LAN clients arriving through the VIP.
  #   - `https://plex.home.keithjackson.dev:32400` -> nothing listens. The Gateway serves this host on
  #     443; measured `http=000` against 192.168.2.20:32400.
  # With every advertised connection dead, plex.tv had nothing publishable, sat at
  # `mapping state 'Mapped - Not Published'`, fell back to Plex Relay, and THE SERVER DISAPPEARED
  # FROM THE ACCOUNT — clients reported "not authorized".
  #
  # DECLARING IT HERE rather than in the web UI is the durable half of the fix. The image's
  # /etc/cont-init.d/40-plex-first-run runs `setPref "customConnections" "${ADVERTISE_IP}"` (line 127)
  # OUTSIDE the PLEX_CLAIM/PlexOnlineToken guard, and its `/.firstRunComplete` marker sits on the
  # container's OVERLAY fs, not the /config PVC — so this is re-applied on EVERY pod start and git,
  # not a form field, owns the value. Editing this env rolls the Deployment, which is what applies it.
  #
  # TWO LIMITS, stated rather than discovered later:
  #   - THE HASH IS SERVER IDENTITY, which is why the whole URL is a cluster-var and not built from
  #     ${PLEX_L4_VIP}: envsubst cannot dash an IP, and the hash is not derivable from anything in
  #     this tree. It is not a secret (it ships in the cert and in the published connection list), but
  #     if Plex reissues under a new hash this URL goes stale and must be re-read from the live cert.
  #   - THIS RESTORES LAN CLIENTS AND THE ACCOUNT LISTING, NOT WAN. Remote clients still need the
  #     off-repo tcp/32400 port-forward (runbooks/plex-remote-access.md). Once that exists, the WAN
  #     twin of this URL can be appended comma-separated — customConnections takes a list.
  - name: ADVERTISE_IP
    value: ${PLEX_ADVERTISE_URL}
# HARDWARE TRANSCODING — the host iGPU's render device (schema 1.8.0 `hardware`; R-PLAT-02).
# INTENT ONLY, AND THE MECHANISM DELIBERATELY DOES NOT LIVE HERE. This says "this workload needs an
# Intel Quick Sync render device"; it does NOT name a device path and it does NOT name a resource.
# How the platform hands the device over is `catalog/render.py`'s business and is documented in
# `schemas/renderer-contract.md` `### hardware` — read that for the mechanism, never this comment.
# The payoff is proven, not theoretical: on 2026-10-01 the mechanism was swapped outright (hostPath
# -> Intel GPU device plugin extended resource) and THIS BLOCK DID NOT MOVE — a `render.py` change
# plus a re-render, zero catalog edits. Do not "simplify" it into a raw `/dev/...` path; that is the
# whole point of the enum.
#
# THE DEVICE IS PRESENT ON EVERY PROD NODE, so nothing had to be installed and Plex can land
# anywhere. Measured on all three nodes 2026-09-30: `/dev/dri/renderD128` present, `root:render`,
# GID 992, mode 0660, driver `i915` bound (NOT `xe`), UHD Graphics 630 on an i7-9700T, media engines
# `vcs0`/`vecs0` live, iGPU enabled in BIOS on each. Measured again 2026-10-01, one layer up: all
# three nodes advertise `gpu.intel.com/i915: 1` in `.status.allocatable`. This field renders no
# nodeSelector; placement is a separate concern with its own justification.
#
# PLEX'S SCHEDULABILITY IS NOW COUPLED TO THE `intel-gpu-plugin` DaemonSet, and that is a real cost,
# not a footnote. The current mechanism is a hard scheduling constraint: if the resource is
# unavailable (plugin broken, removed, or not yet reconciled) this pod is UNSCHEDULABLE, and because
# `apps-wave-0` is `wait: true` with `apps-wave-1` `dependsOn` it, that STALLS THE WHOLE APP TIER.
# The previous hostPath mechanism degraded silently to software transcoding instead — but it never
# actually granted access (device-cgroup EPERM, measured 2026-10-01), so there was nothing to keep.
# THE ESCAPE HATCH IS THIS BLOCK: delete these two lines and re-render, and Plex goes back to
# software transcoding and the tier unblocks. One field, no code change. `runbooks/plex-quicksync.md`
# carries that procedure for whoever needs it at 2am.
#
# NO `podSecurityContext.supplementalGroups`, AND ADDING ONE WOULD BE INERT — do not "fix" this.
# The device is `root:render` 0660 and this process is uid/gid 1000, so the obvious move is to push
# the host `render` GID in as a supplemental group. It cannot work here, and this survived the
# mechanism swap unchanged. MEASURED IN THE RUNNING CONTAINER, 2026-09-30 and again 2026-10-01: the
# image's own `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` stats every char device
# under `/dev/dri`, `groupadd`s a matching group INSIDE the container when none exists, and
# `usermod -a -G`s the `plex` user into it — every start, nothing hardcoded, which also makes a host
# GID renumbering self-healing (no GID is pinned anywhere in this repo) and which works the same way
# on a PLUGIN-INJECTED device as on a bind-mounted one. Measured live, `Plex Media Server` ran with
# `Groups: 44 100 992 1000` — 992 IS the host `render` GID, i.e. the hook had already done the job.
# The service execs via `s6-setuidgid plex`, and `s6-setuidgid` REPLACES the process group list from
# `/etc/group` rather than appending to it, so root's gid `0` — which PID 1 (`s6-svscan`) does hold —
# is GONE, and anything Kubernetes supplies reaches s6-svscan and is discarded before Plex ever
# starts. The group is the image's job and it does it correctly. `render.py --selftest` asserts the
# ABSENCE of a pod securityContext for this field precisely so nobody adds an inert one back later.
#
# THE LAST MILE IS A MANUAL OPERATOR STEP AND CANNOT LIVE IN THIS REPO. Handing over the device does
# not enable hardware transcoding — Plex's own `HardwareAcceleratedCodecs` preference does, and it
# is ABSENT from `Preferences.xml` today (the toggle has never been on). NO environment variable can
# set it: a grep of the running image's entire init surface (`/etc/cont-init.d`, `/etc/services.d`,
# `/etc/fix-attrs.d`, `/plex-common.sh`, `/usr/local/bin`) found ZERO `PLEX_PREFERENCE*` support,
# and the six prefs the image does write (`MachineIdentifier`, `ProcessedMachineIdentifier`,
# `PlexOnlineToken`, `customConnections`, `allowedNetworks`, `TranscoderTempDirectory`) are all
# unrelated. So there is no GitOps path to the toggle; an operator clicks it once in the web UI.
# `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took.
hardware:
  gpu: intel-quicksync
exposure: public              # edge port-forward + public DNS; clients + Seerr reach it (docs/09; R-DNS/R-NET)
egress: internet              # normal WAN egress (docs/03; R-NET-05)
storage:
  - name: config
    size: 8Gi
    class: longhorn           # Plex DB/metadata: replicated app state on the 3-replica tier (docs/04; R-STATE-01/02)
    mountPath: /config        # plexinc/pms-docker writes its identity/claim + "Library" DB under /config, NOT the
                              # renderer default /data/config. Without this the PVC sat unused and Plex wrote to the
                              # container's EPHEMERAL layer -> unclaimed + all state lost on restart. The image inits as
                              # root (s6-svscan PID 1), chowns /config to the plex user, then drops -> no fsGroup needed
                              # (verified in-cluster: PID1=root, "Plex Media Server"=plex). Persisting /config here also
                              # makes the claim survive restarts once claimed — which is what makes the PLEX_CLAIM env
                              # above a ONE-SHOT: the token is exchanged for a PlexOnlineToken written here, and the
                              # exchange is then skipped on every later start. (The older note here claimed a
                              # placeholder claim Secret "would crashloop the pod"; upstream's first-run script says
                              # otherwise — see the PLEX_CLAIM comment in `env` above. An ABSENT Secret is the hard
                              # failure, and that is why one is committed.)
  # THE LIBRARY (docs/04 bulk tier). One READ-ONLY mount of the export ROOT, giving /media/movies,
  # /media/tv and /media/photos — the whole tree the layout actually has (`ls /mnt/Storage/plex`:
  # downloads/ movies/ photos/ tv/).
  #
  # WHY THE ROOT AND NOT THREE subPath MOUNTS. Nothing is gained by narrowing: the mount is `ro`,
  # so no subPath could remove a privilege that matters, while three mounts would triple the NFS
  # mounts per pod and mean a manifest change every time a top-level library is added.
  #
  # WHY READ-ONLY, when the export is `rw` and the other consumers write. Plex is a SERVER: it
  # indexes and streams. Every write on that tree belongs to Sonarr and Radarr. Marking it here
  # costs nothing and removes Plex from the set of things that can damage 8.33 T. Enforced by the
  # PV's nfs.readOnly (a real `ro` mount), not only by the volumeMount flag.
  #
  # UID: no change needed. The tree is 0755 owned 1000:3000 throughout, so it is world-readable and
  # world-traversable, and PLEX_UID/PLEX_GID above already make the process uid 1000 in any case.
  # (fsGroup would have been a no-op here regardless — the in-tree NFS plugin does no ownership
  # management. That is the trap the arrs' PUID/PGID exists to avoid; Plex never hits it.)
  - name: media
    class: bulk
    export: /mnt/Storage/plex   # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER}
    mountPath: /media
    readOnly: true
    size: 9Ti                   # ADVISORY ONLY (nothing enforces a static NFS PV's capacity): the
                                # dataset's 8.33 T used, rounded up, so `kubectl get pv` reports
                                # something true. The real limit is the ZFS pool.
dependencies: []              # default-deny: reaches only DNS; clients/Seerr reach IT (docs/03; R-NET-01/02)
ingressRoutes:
  - host: plex.${BASE_DOMAIN}
    port: 32400               # Plex web UI + API served on 32400 (wildcard TLS at the Gateway; docs/08/09)
auth:
  mode: native                # Plex account auth — documented exception, NOT Keycloak (docs/13/15; R-IDP-02)
update:
  channel: stable             # R-UPD-01 (never pre-release)
  range: ">=1.40.0"           # acceptable versions (advisory; image-automation off during onboarding)
  cadence: "0 4 * * 0"        # weekly stable check/apply
  autoRollbackOnFailedHealth: true   # R-UPD-03
observability:
  metrics: true
  slis:
    - up                      # R-OBS-01: service-down alert
backup:
  # The config volume is on the longhorn (replicated, "irreplaceable") tier, so the L0 Kyverno
  # backup policy (R-STATE-03) forbids `reacquirable` here — it must reference an off-cluster
  # schedule. Plex's metadata DB (library index, thumbnails, watch state) is expensive to rebuild,
  # so it is backed up off-cluster. The bulk library itself, once attached, is the `reacquirable`
  # part (docs/04) and is NOT covered by this schedule.
  policy: daily-offcluster    # off-cluster backup of the longhorn config volume (docs/04; R-STATE-03)