Merge pull request 'feat(plex): one advertised hostname for local+remote, its own LE cert, and a LAN definition so LAN clients stop reading as remote' (#48) from feat/plex-vip-hostname-and-lan-networks into main

Reviewed-on: #48
Reviewed-by: Keith Jackson <[email protected]>

catalog/pihole.yaml

@@ -134,8 +134,17 @@ env:
   # then silently wrong: FTL stores `["a","b"]` as ONE array element containing those literal
   # characters, writes that single bogus line into /etc/pihole/dnsmasq.conf and reports "0 invalid"
   # while resolution for the zone does not work. Confirmed both ways against the live image.
+  # plex-vip -> ${PLEX_L4_VIP} (added 2026-10-04) is the LAN HALF of Plex's single advertised
+  # hostname. Plex publishes ONE URL to plex.tv for both local and remote clients
+  # (catalog/plex.yaml ADVERTISE_IP): on the LAN this answer points it at the tcp/32400
+  # LoadBalancer, while Cloudflare answers the same name publicly with the WAN address, NATed to
+  # the same VIP. WITHOUT THIS LINE the entry above answers it with ${GATEWAY_VIP} via the zone
+  # wildcard, where NOTHING listens on 32400 — measured 2026-10-04, plex-vip.${BASE_DOMAIN}
+  # resolved to the Gateway VIP and a connect to :32400 there returned http=000. That is a silent
+  # failure: the Plex web UI on plex.${BASE_DOMAIN} keeps working through the Gateway on 443 while
+  # every client that takes the ADVERTISED connection cannot reach the server at all.
   - name: FTLCONF_misc_dnsmasq_lines
-    value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}"
+    value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
   # THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
   # apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
   # carries the full rationale and the measurement that found it.

catalog/plex.yaml

@@ -72,54 +72,211 @@ env:
       secretKeyRef:
         name: plex-claim
         key: PLEX_CLAIM
-  # ADVERTISE_IP ("Custom server access URLs") — NOW SET. This replaces a DEFERRAL note that stood
-  # here until 2026-10-04, and the deferral's premise has changed: it weighed only ONE candidate,
-  # `https://plex.home.keithjackson.dev` through the L7 Gateway. The operator chose the NATIVE L4
-  # path instead on 2026-10-03 (apps/wave-0/plex-remote.yaml) so streams go straight to
-  # ${PLEX_L4_VIP}:32400 and bypass the Envoy hop. What was still missing was a URL whose TLS
-  # CERTIFICATE validates — and getting that wrong is what took the service down.
+  # ADVERTISE_IP ("Custom server access URLs") — the ONE hostname this server publishes to plex.tv,
+  # serving BOTH local and remote clients by split-horizon DNS (operator design, 2026-10-04):
+  #   LAN    -> PiHole answers plex-vip.${BASE_DOMAIN} with ${PLEX_L4_VIP} (the tcp/32400 LoadBalancer
+  #             in apps/wave-0/plex-remote.yaml). See catalog/pihole.yaml's split-horizon line, which
+  #             needs a per-host override because the zone wildcard otherwise points at the GATEWAY
+  #             VIP — measured 2026-10-04: plex-vip.${BASE_DOMAIN} resolved to the Gateway, where
+  #             nothing listens on 32400.
+  #   PUBLIC -> Cloudflare answers with the WAN address, NATed to the same VIP (off-repo).
   #
-  # THE VALUE IS A `plex.direct` NAME, NOT THE RAW VIP, AND THAT IS THE WHOLE POINT. Plex serves
-  # 32400 with its own certificate; measured live 2026-10-04 it carries exactly one name:
-  #     CN = *.<serverhash>.plex.direct      (one SAN, no IP SANs at all)
-  # so `https://<ip>:32400` can NEVER validate. `<dashed-ip>.<serverhash>.plex.direct` is Plex's own
-  # mechanism for this: it is PUBLIC DNS that resolves to the embedded address, so the name points at
-  # the LAN VIP while the wildcard cert above matches it. Measured 2026-10-04 with FULL verification
-  # (no `curl -k`):
-  #     https://192-168-2-24.<hash>.plex.direct:32400/identity -> 200, TLS verify OK
-  #     https://192.168.2.24:32400/identity                    -> 000, cert validation FAILED
+  # WHY A HOSTNAME AND NOT THE `plex.direct` FORM THIS BRIEFLY USED. The first fix for the
+  # 2026-10-04 outage advertised `https://<dashed-ip>.<serverhash>.plex.direct:32400`, which works
+  # (and is Plex's own mechanism), but it hard-codes BOTH the VIP and the server's certificate hash
+  # into a cluster-var — two values nothing could keep in sync, with no alert if either drifted. A
+  # hostname in our own zone removes that whole class of problem: ${BASE_DOMAIN} is already a knob,
+  # the VIP lives only in PiHole's answer, and `PLEX_ADVERTISE_URL` is deleted outright.
   #
-  # THIS FIELD EXISTS BECAUSE AUTO-DISCOVERY CANNOT WORK HERE. Left empty, Plex publishes the
-  # addresses it can see itself — which from a pod is the overlay IP (`10.0.2.136`), useless to a LAN
-  # client. The VIP has to be advertised explicitly; there is no "just clear it" fix.
+  # THIS IS `https`, WHICH ONLY WORKS BECAUSE WE HAND PLEX OUR OWN CERTIFICATE. Plex serves 32400
+  # with its `*.<serverhash>.plex.direct` cert, which does NOT cover this name — advertising
+  # `https://plex-vip...` against the stock cert reproduces the exact failure that caused the
+  # outage (measured: a storm of `CERT: incomplete TLS handshake ... certificate unknown` from real
+  # clients, and `curl` without `-k` failing to connect). The `lifecycle.postStart` hook below
+  # installs the Let's Encrypt cert for this name and points Plex at it, so the advertised URL and
+  # the served certificate agree. Do not change this to a bare IP, and do not drop the hook and
+  # leave the `https` here.
   #
-  # WHAT WENT WRONG, recorded so it is not re-introduced by hand. customConnections OVERRIDES Plex's
-  # own publishing, and on 2026-10-04 it held two hand-typed URLs, both uncompletable:
-  #   - `https://192.168.2.24:32400`              -> cert cannot cover an IP. Measured: a storm of
-  #     `CERT: incomplete TLS handshake ... sslv3 alert certificate unknown` in Plex's log every few
-  #     seconds, from REAL LAN clients arriving through the VIP.
-  #   - `https://plex.home.keithjackson.dev:32400` -> nothing listens. The Gateway serves this host on
-  #     443; measured `http=000` against 192.168.2.20:32400.
-  # With every advertised connection dead, plex.tv had nothing publishable, sat at
-  # `mapping state 'Mapped - Not Published'`, fell back to Plex Relay, and THE SERVER DISAPPEARED
-  # FROM THE ACCOUNT — clients reported "not authorized".
+  # (A SAN match is not what makes Plex *work* — `httpsRequired=0` and `secureConnections` default
+  # to Preferred, so clients can fall back to cleartext. It is what keeps the remote leg ENCRYPTED
+  # instead of silently downgrading traffic that crosses the internet.)
   #
-  # DECLARING IT HERE rather than in the web UI is the durable half of the fix. The image's
-  # /etc/cont-init.d/40-plex-first-run runs `setPref "customConnections" "${ADVERTISE_IP}"` (line 127)
-  # OUTSIDE the PLEX_CLAIM/PlexOnlineToken guard, and its `/.firstRunComplete` marker sits on the
-  # container's OVERLAY fs, not the /config PVC — so this is re-applied on EVERY pod start and git,
-  # not a form field, owns the value. Editing this env rolls the Deployment, which is what applies it.
-  #
-  # TWO LIMITS, stated rather than discovered later:
-  #   - THE HASH IS SERVER IDENTITY, which is why the whole URL is a cluster-var and not built from
-  #     ${PLEX_L4_VIP}: envsubst cannot dash an IP, and the hash is not derivable from anything in
-  #     this tree. It is not a secret (it ships in the cert and in the published connection list), but
-  #     if Plex reissues under a new hash this URL goes stale and must be re-read from the live cert.
-  #   - THIS RESTORES LAN CLIENTS AND THE ACCOUNT LISTING, NOT WAN. Remote clients still need the
-  #     off-repo tcp/32400 port-forward (runbooks/plex-remote-access.md). Once that exists, the WAN
-  #     twin of this URL can be appended comma-separated — customConnections takes a list.
+  # The image's /etc/cont-init.d/40-plex-first-run runs `setPref "customConnections"
+  # "${ADVERTISE_IP}"` (line 127, verified in the running image), OUTSIDE the
+  # PLEX_CLAIM/PlexOnlineToken guard, and its `/.firstRunComplete` marker is on the container
+  # OVERLAY fs rather than the /config PVC — so this is re-applied on EVERY pod start and git, not
+  # a web form, owns the value. Hand-typed custom URLs are what broke this service; they are now
+  # overwritten on every start.
   - name: ADVERTISE_IP
-    value: ${PLEX_ADVERTISE_URL}
+    value: https://plex-vip.${BASE_DOMAIN}:32400
+  # THE TLS MATERIAL FOR THE NAME ABOVE, as PEM text in env rather than a mounted volume.
+  # `plex-vip-tls` is issued by cert-manager from ClusterIssuer `letsencrypt-prod` for exactly
+  # `plex-vip.${BASE_DOMAIN}` (apps/wave-0/plex-cert.yaml, same wave => same reconcile as this
+  # workload). The postStart hook converts it to the PKCS#12 bundle Plex requires.
+  #
+  # WHY ENV AND NOT A VOLUME, which would be the conventional choice. MEASURED 2026-10-04:
+  # `schemas/servicedefinition.schema.json` `storage` accepts only
+  # [class, export, mountPath, name, readOnly, size, subPath] with `additionalProperties: false`,
+  # `catalog/render.py` emits NO secret volumes, and no rendered app mounts a Secret as a volume.
+  # Mounting one therefore needs a schema + renderer change; `env.valueFrom.secretKeyRef` is
+  # already in the schema and already used by PLEX_CLAIM above. PEM is text, so it survives an env
+  # var intact (a raw PKCS#12 would not — it is binary and carries NUL bytes).
+  #
+  # COST, STATED PLAINLY: the private key is readable from this container's environment
+  # (/proc/1/environ, `kubectl exec env`) by anything that can already exec into the pod. That is
+  # the same audience that can read the converted bundle under /config, so the marginal exposure is
+  # small — but a secret VOLUME would still be the better shape, and adding secret-volume support
+  # to the schema is the clean follow-up that removes this note.
+  #
+  # `optional: true` IS LOAD-BEARING, NOT DEFENSIVE CLUTTER (schema 1.11.0, added for this). The
+  # Certificate is issued INTO THIS NAMESPACE, and the namespace is created by the rendered
+  # apps/wave-0/plex.yaml itself — so the cert CANNOT be issued in an earlier tier, and the
+  # Certificate and this Deployment land in the SAME reconcile. On a cold bootstrap the Secret is
+  # therefore briefly absent while cert-manager completes a DNS01 challenge. With a HARD ref, Plex
+  # would sit in CreateContainerConfigError until a certificate issued — and because `apps-wave-0`
+  # is `wait: true` with `apps-wave-1` dependsOn it, a failed or slow issuance would STALL THE WHOLE
+  # APP TIER (the same hazard the `hardware` block below documents for the GPU resource). Optional
+  # instead means: no cert -> Plex still starts and serves, the postStart hook logs that the
+  # certificate was not installed, and the next start after issuance converges. Plex falls back to
+  # its own `*.plex.direct` certificate in the meantime, so the degraded state is "advertised URL
+  # not yet servable over TLS", not "media server down".
+  - name: PLEX_TLS_CRT
+    valueFrom:
+      secretKeyRef:
+        name: plex-vip-tls
+        key: tls.crt
+        optional: true
+  - name: PLEX_TLS_KEY
+    valueFrom:
+      secretKeyRef:
+        name: plex-vip-tls
+        key: tls.key
+        optional: true
+lifecycle:
+  # LEVEL-TRIGGERED PLEX SETTINGS, re-asserted on EVERY container start — the same pattern and the
+  # same reasoning as catalog/pihole.yaml's and catalog/qbittorrent.yaml's postStart: desired state
+  # lives in git and is re-asserted unconditionally rather than typed into a web form once. Typing
+  # settings into the web form is precisely what took this service down on 2026-10-04.
+  #
+  # IT EXISTS BECAUSE THE IMAGE HAS NO ENV FOR THESE SETTINGS. Measured 2026-10-04 across the whole
+  # init surface (/etc/cont-init.d, /etc/services.d, /plex-common.sh, /usr/local/bin): the image
+  # calls setPref for exactly SIX prefs — MachineIdentifier, ProcessedMachineIdentifier,
+  # PlexOnlineToken, customConnections, allowedNetworks, TranscoderTempDirectory — and supports NO
+  # generic PLEX_PREFERENCE* hook of any kind. The two things we need (a LAN definition and a custom
+  # certificate) are reachable ONLY through Plex's own API, so this hook is the only GitOps path.
+  #
+  # IT MUST NEVER FAIL. A postStart hook that exits non-zero makes kubelet KILL the container, so
+  # every step is bounded and the script ends with an unconditional `exit 0`. A missing token, an
+  # unissued certificate or an unreachable API degrades to "not applied this start" and is retried
+  # on the next one; it never crashloops Plex.
+  postStart:
+    exec:
+      command:
+        - /bin/bash
+        - -c
+        - |
+          # NOTE — LOAD-BEARING: SHELL VARIABLES HERE ARE WRITTEN BARE, NEVER BRACE-WRAPPED.
+          # This whole script is rewritten by Flux postBuild envsubst BEFORE it ever runs, so a
+          # brace-wrapped name is read as a cluster-vars knob: an undeclared one is a HARD
+          # Kustomization failure under StrictPostBuildSubstitutions, and a declared one would be
+          # replaced with its value. Shell vars are therefore `$PX`, `$TOKEN`, `$P12` — bare. Only
+          # BASE_DOMAIN below is brace-wrapped, because it IS a knob and IS meant to be
+          # substituted. Command substitution `$(cmd)` is left alone and needs no escaping. A shell
+          # variable written with braces would be blanked or would fail the whole Kustomization —
+          # this repo has already lost a gitea OIDC bootstrap to exactly that mistake, and
+          # catalog/render.py now refuses to emit it (which is how this very comment got caught).
+          # DELIBERATELY NO `set -u`. The TLS env vars are declared `optional: true`, so when the
+          # certificate has not issued yet they are genuinely UNSET, not empty — and under `set -u`
+          # the first `[ -n "$PLEX_TLS_CRT" ]` would abort the script, which for a postStart hook
+          # means kubelet KILLS THE CONTAINER. That is the precise failure the `optional` flag exists
+          # to prevent, so turning it back on here would defeat the whole design. (The alternative,
+          # a brace-wrapped `:-` default, would need envsubst escaping — the trap the note above
+          # describes.) Every value below is explicitly guarded instead.
+          PX="/config/Library/Application Support/Plex Media Server/Preferences.xml"
+          CERTDIR=/config/certs
+          P12=$CERTDIR/plex-vip.p12
+          # The PKCS#12 passphrase protects nothing and is deliberately not a secret: Plex needs it
+          # in cleartext in customCertificateKey, and it and the bundle sit on the same PVC. It is a
+          # container-format requirement, not an access control.
+          P12PASS=plexvip
+          LANNETS='10.0.0.0/8,192.168.0.0/16'
+          CERTDOMAIN=plex-vip.${BASE_DOMAIN}
+          API=http://127.0.0.1:32400
+
+          # (1) Build the PKCS#12 bundle from the PEM the Secret supplies. Rebuilt on every start so
+          # a cert-manager RENEWAL is picked up with no extra moving part.
+          if [ -n "$PLEX_TLS_CRT" ] && [ -n "$PLEX_TLS_KEY" ]; then
+            mkdir -p $CERTDIR && chmod 700 $CERTDIR
+            printf '%s' "$PLEX_TLS_CRT" > $CERTDIR/tls.crt
+            printf '%s' "$PLEX_TLS_KEY" > $CERTDIR/tls.key
+            chmod 600 $CERTDIR/tls.key
+            if openssl pkcs12 -export -out $P12.tmp -inkey $CERTDIR/tls.key \
+                 -in $CERTDIR/tls.crt -passout pass:$P12PASS 2>/dev/null; then
+              mv -f $P12.tmp $P12
+              chown 1000:1000 $P12 2>/dev/null || true
+              chmod 600 $P12
+              echo "[plex-settings] PKCS#12 bundle written"
+            else
+              rm -f $P12.tmp
+              echo "[plex-settings] WARN: pkcs12 export failed; keeping any previous bundle" >&2
+            fi
+            rm -f $CERTDIR/tls.key
+          else
+            echo "[plex-settings] WARN: TLS env empty; certificate not installed this start" >&2
+          fi
+
+          # (2) Wait, bounded (~90s), for the local API. /identity needs no auth.
+          i=0
+          while [ "$i" -lt 45 ]; do
+            if [ "$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 $API/identity 2>/dev/null)" = "200" ]; then
+              break
+            fi
+            i=$(expr $i + 1); sleep 2
+          done
+
+          # (3) The prefs API needs the server token. It already lives in Preferences.xml on this
+          # PVC; it is never echoed here, and Plex redacts tokens in its own request log. An
+          # UNCLAIMED server has none — a normal state on a fresh /config — so skip, do not fail.
+          TOKEN=$(grep -o 'PlexOnlineToken="[^"]*"' "$PX" 2>/dev/null | head -1 | cut -d'"' -f2)
+          if [ -z "$TOKEN" ]; then
+            echo "[plex-settings] not claimed yet — skipping prefs, will retry next start" >&2
+            exit 0
+          fi
+
+          # Percent-encode what our values actually contain. Plex takes prefs as QUERY parameters on
+          # a PUT, so a raw '/' or ',' would be read as structure rather than data.
+          enc() {
+            printf '%s' "$1" | sed -e 's|%|%25|g' -e 's|/|%2F|g' -e 's|,|%2C|g' \
+                                   -e 's|:|%3A|g' -e 's| |%20|g'
+          }
+          setpref() {
+            code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 -X PUT \
+              -H "X-Plex-Token: $TOKEN" "$API/:/prefs?$1=$(enc "$2")" 2>/dev/null)
+            if [ "$code" = "200" ]; then
+              echo "[plex-settings] set $1"
+            else
+              echo "[plex-settings] WARN: $1 -> HTTP $code" >&2
+            fi
+          }
+
+          # (4) THE LAN DEFINITION — the reason this hook was written. Without it Plex has NO notion
+          # of a local network and tags EVERY client remote. Measured 2026-10-04: 468 requests
+          # logged "Allowed Network (WAN)" and ZERO "(LAN)", including Seerr from inside the
+          # cluster, which is why the server showed as Remote to LAN clients. Note allowedNetworks
+          # (the ALLOWED_NETWORKS env) only waives AUTH — it does NOT make a client local. Two
+          # different settings; only one of them has an env hook.
+          setpref LanNetworksBandwidth "$LANNETS"
+
+          # (5) THE CERTIFICATE, which is what makes the `https` in ADVERTISE_IP servable. Set only
+          # if the bundle exists — pointing Plex at a missing file would break its TLS listener.
+          if [ -s "$P12" ]; then
+            setpref customCertificatePath "$P12"
+            setpref customCertificateKey "$P12PASS"
+            setpref customCertificateDomain "$CERTDOMAIN"
+          else
+            echo "[plex-settings] WARN: no bundle; leaving certificate prefs untouched" >&2
+          fi
+          exit 0
 # HARDWARE TRANSCODING — the host iGPU's render device (schema 1.8.0 `hardware`; R-PLAT-02).
 # INTENT ONLY, AND THE MECHANISM DELIBERATELY DOES NOT LIVE HERE. This says "this workload needs an
 # Intel Quick Sync render device"; it does NOT name a device path and it does NOT name a resource.

catalog/render.py

@@ -2209,6 +2209,23 @@ EXTRA_UNITS: dict = {
             "plex-remote.yaml",
         ],
     },
+    # Plex's own TLS certificate for the single hostname it advertises to plex.tv
+    # (apps/wave-0/plex-cert.yaml). `above: False` co-locates it in plex's OWN wave, which is the
+    # only placement available: the Certificate must land in the `plex` namespace, and that
+    # namespace is created by the rendered plex.yaml in this very wave, so it CANNOT be issued
+    # earlier. The workload tolerates the resulting gap by marking the env refs `optional` rather
+    # than by ordering -- see the header of plex-cert.yaml.
+    #
+    # Non-catalog because the ServiceDefinition schema has no field for a cert-manager Certificate
+    # (additionalProperties is false); `ingressRoutes` inherits the Gateway wildcard instead, which
+    # is the opposite case to a workload terminating its own TLS on a raw L4 port.
+    "plex-cert": {
+        "anchor": "plex",
+        "above": False,
+        "files": [
+            "plex-cert.yaml",
+        ],
+    },
 }