From 1f4552d40906b927f415cf87801a82f451d04be5 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 4 Oct 2026 02:53:01 -0500 Subject: [PATCH] fix(plex): advertise the plex.direct name, not the bare VIP (server vanished from the plex.tv account) customConnections held two hand-typed URLs and neither could ever complete, so Plex had nothing publishable and the server disappeared from the account: - https://192.168.2.24:32400 -- the server's cert is *.130072f20c334d6c8f63d0dfef87309b.plex.direct with NO IP SAN, so bare-IP https fails TLS for every client. Measured: a continuous storm of "CERT: incomplete TLS handshake ... certificate unknown" from real LAN clients arriving through the plex-l4-tcp VIP. - https://plex.home.keithjackson.dev:32400 -- nothing listens; the Gateway serves that host on 443 (measured http=000 against 192.168.2.20:32400). With no completable connection plex.tv published nothing ("Mapped - Not Published"), Plex fell back to Relay, and clients reported "not authorized". Fix: advertise ..plex.direct -- public DNS that resolves to the embedded address, so it points at the LAN VIP and matches the wildcard cert. Verified with full TLS validation (no curl -k): 200 + verify OK, where the bare-IP form fails to connect. Declared as ADVERTISE_IP so git owns the value instead of a web form: the image's 40-plex-first-run writes it into customConnections on every pod start, outside the claim guard. Does NOT fix off-LAN access -- the advertised address is RFC1918 and the WAN tcp/32400 forward remains unverified. ManualPortMappingMode=1 with an absent port is left as-is (no GitOps path exists for it). Touches clusters/** -> sensitive paths -> needs the operator's /approve-override, deliberately not self-posted (CLAUDE.md 1). Co-Authored-By: Claude Opus 5 (1M context) --- apps/wave-0/plex.yaml | 2 + catalog/plex.yaml | 56 ++++++- clusters/base/cluster-vars.yaml | 28 +++- clusters/prod/kustomization.yaml | 10 ++ .../2026-10-04-plex-advertise-plexdirect.md | 139 ++++++++++++++++++ 5 files changed, 226 insertions(+), 9 deletions(-) create mode 100644 decisions/2026-10-04-plex-advertise-plexdirect.md diff --git a/apps/wave-0/plex.yaml b/apps/wave-0/plex.yaml index a374ec6..e4c3d3e 100644 --- a/apps/wave-0/plex.yaml +++ b/apps/wave-0/plex.yaml @@ -137,6 +137,8 @@ spec: secretKeyRef: name: plex-claim key: PLEX_CLAIM + - name: ADVERTISE_IP + value: ${PLEX_ADVERTISE_URL} resources: limits: gpu.intel.com/i915: 1 diff --git a/catalog/plex.yaml b/catalog/plex.yaml index 85b697e..7f75d14 100644 --- a/catalog/plex.yaml +++ b/catalog/plex.yaml @@ -72,14 +72,54 @@ env: secretKeyRef: name: plex-claim key: PLEX_CLAIM - # ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex - # which URL to advertise to plex.tv for REMOTE clients. The only candidate is - # https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL - # Gateway VIP (conformance/tests/l2-behavioral/dns-internal-resolution), and both its public - # reachability and whether the TLS-terminating Cilium Gateway proxies Plex's 32400 API correctly for - # remote access are UNVERIFIED. Advertising a wrong/internal-only URL breaks remote access for ALL - # clients, so per "prefer safety over completeness" it is left unset. FOLLOW-UP: once remote - # reachability through the Gateway is confirmed, set ADVERTISE_IP: "https://plex.home.keithjackson.dev:443". + # ADVERTISE_IP ("Custom server access URLs") — NOW SET. This replaces a DEFERRAL note that stood + # here until 2026-10-04, and the deferral's premise has changed: it weighed only ONE candidate, + # `https://plex.home.keithjackson.dev` through the L7 Gateway. The operator chose the NATIVE L4 + # path instead on 2026-10-03 (apps/wave-0/plex-remote.yaml) so streams go straight to + # ${PLEX_L4_VIP}:32400 and bypass the Envoy hop. What was still missing was a URL whose TLS + # CERTIFICATE validates — and getting that wrong is what took the service down. + # + # THE VALUE IS A `plex.direct` NAME, NOT THE RAW VIP, AND THAT IS THE WHOLE POINT. Plex serves + # 32400 with its own certificate; measured live 2026-10-04 it carries exactly one name: + # CN = *..plex.direct (one SAN, no IP SANs at all) + # so `https://:32400` can NEVER validate. `..plex.direct` is Plex's own + # mechanism for this: it is PUBLIC DNS that resolves to the embedded address, so the name points at + # the LAN VIP while the wildcard cert above matches it. Measured 2026-10-04 with FULL verification + # (no `curl -k`): + # https://192-168-2-24..plex.direct:32400/identity -> 200, TLS verify OK + # https://192.168.2.24:32400/identity -> 000, cert validation FAILED + # + # THIS FIELD EXISTS BECAUSE AUTO-DISCOVERY CANNOT WORK HERE. Left empty, Plex publishes the + # addresses it can see itself — which from a pod is the overlay IP (`10.0.2.136`), useless to a LAN + # client. The VIP has to be advertised explicitly; there is no "just clear it" fix. + # + # WHAT WENT WRONG, recorded so it is not re-introduced by hand. customConnections OVERRIDES Plex's + # own publishing, and on 2026-10-04 it held two hand-typed URLs, both uncompletable: + # - `https://192.168.2.24:32400` -> cert cannot cover an IP. Measured: a storm of + # `CERT: incomplete TLS handshake ... sslv3 alert certificate unknown` in Plex's log every few + # seconds, from REAL LAN clients arriving through the VIP. + # - `https://plex.home.keithjackson.dev:32400` -> nothing listens. The Gateway serves this host on + # 443; measured `http=000` against 192.168.2.20:32400. + # With every advertised connection dead, plex.tv had nothing publishable, sat at + # `mapping state 'Mapped - Not Published'`, fell back to Plex Relay, and THE SERVER DISAPPEARED + # FROM THE ACCOUNT — clients reported "not authorized". + # + # DECLARING IT HERE rather than in the web UI is the durable half of the fix. The image's + # /etc/cont-init.d/40-plex-first-run runs `setPref "customConnections" "${ADVERTISE_IP}"` (line 127) + # OUTSIDE the PLEX_CLAIM/PlexOnlineToken guard, and its `/.firstRunComplete` marker sits on the + # container's OVERLAY fs, not the /config PVC — so this is re-applied on EVERY pod start and git, + # not a form field, owns the value. Editing this env rolls the Deployment, which is what applies it. + # + # TWO LIMITS, stated rather than discovered later: + # - THE HASH IS SERVER IDENTITY, which is why the whole URL is a cluster-var and not built from + # ${PLEX_L4_VIP}: envsubst cannot dash an IP, and the hash is not derivable from anything in + # this tree. It is not a secret (it ships in the cert and in the published connection list), but + # if Plex reissues under a new hash this URL goes stale and must be re-read from the live cert. + # - THIS RESTORES LAN CLIENTS AND THE ACCOUNT LISTING, NOT WAN. Remote clients still need the + # off-repo tcp/32400 port-forward (runbooks/plex-remote-access.md). Once that exists, the WAN + # twin of this URL can be appended comma-separated — customConnections takes a list. + - name: ADVERTISE_IP + value: ${PLEX_ADVERTISE_URL} # HARDWARE TRANSCODING — the host iGPU's render device (schema 1.8.0 `hardware`; R-PLAT-02). # INTENT ONLY, AND THE MECHANISM DELIBERATELY DOES NOT LIVE HERE. This says "this workload needs an # Intel Quick Sync render device"; it does NOT name a device path and it does NOT name a resource. diff --git a/clusters/base/cluster-vars.yaml b/clusters/base/cluster-vars.yaml index 026ee84..c5f15d1 100644 --- a/clusters/base/cluster-vars.yaml +++ b/clusters/base/cluster-vars.yaml @@ -101,7 +101,7 @@ data: # tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db: # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers; - # plex-remote: PLEX_L4_VIP), + # plex-remote: PLEX_L4_VIP; plex: PLEX_ADVERTISE_URL), # velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION, # VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an # UNSUBSTITUTED path would ship the literal string into the cluster. @@ -190,6 +190,32 @@ data: # prod -> 192.168.2.24 (next free after pihole-dns .23 in prod's .20-.30 pool; see clusters/prod) PLEX_L4_VIP: "192.168.1.57" # Plex: tcp/32400 remote access (NOT the web UI, which is L7) + # PLEX_ADVERTISE_URL — the connection URL Plex publishes to plex.tv ("Custom server access URLs", + # the ADVERTISE_IP env in catalog/plex.yaml, which the image writes into the `customConnections` + # preference on every pod start). Read catalog/plex.yaml's ADVERTISE_IP block for the full why; + # the short version is the only thing that matters when editing this line: + # + # IT MUST BE A `..plex.direct` NAME, NEVER A BARE IP. Plex serves 32400 + # with a `*..plex.direct` wildcard certificate and NO IP SAN, so `https://:32400` + # fails TLS validation for every client. That exact mistake is what broke this service on + # 2026-10-04: with no completable connection, plex.tv published nothing, the server vanished from + # the account and clients reported "not authorized". The dashed name is public DNS that resolves + # to the embedded address, so it points at the LAN VIP AND matches the cert. + # + # WHY IT IS A WHOLE URL AND NOT BUILT FROM ${PLEX_L4_VIP}: postBuild substitution is plain + # envsubst — it cannot turn 192.168.2.24 into 192-168-2-24 — and `` is this server's + # certificate identity, derivable from nothing in this tree. Re-read it from the live cert + # (`openssl s_client -connect :32400 | openssl x509 -noout -subject`) if Plex ever reissues. + # + # The dashed IP here MUST track ${PLEX_L4_VIP} above. They are two halves of one address and + # nothing checks that they agree. + # + # dev -> PLACEHOLDER, non-functional. dev is decommissioned and has no claimed Plex server, so + # there is no real hash to put here; the base default only has to be a defined, correctly + # shaped string (StrictPostBuildSubstitutions makes an UNDEFINED var a hard failure). + # prod -> the real 192.168.2.24 + live server hash; see clusters/prod/kustomization.yaml. + PLEX_ADVERTISE_URL: "https://192-168-1-57.0000000000000000000000000000000f.plex.direct:32400" + # The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress # peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable # from `world` is an open resolver and an amplification reflector, so the allow is scoped to this diff --git a/clusters/prod/kustomization.yaml b/clusters/prod/kustomization.yaml index e518e64..0046ed6 100644 --- a/clusters/prod/kustomization.yaml +++ b/clusters/prod/kustomization.yaml @@ -496,6 +496,16 @@ patches: - op: replace path: /data/PLEX_L4_VIP value: "192.168.2.24" + # PLEX_ADVERTISE_URL — the prod VIP above in Plex's `plex.direct` form, which is the ONLY form + # that validates: the server's cert is `*.130072f20c334d6c8f63d0dfef87309b.plex.direct` with no + # IP SAN (read off the live listener 2026-10-04), so a bare `https://192.168.2.24:32400` fails + # TLS for every client — the 2026-10-04 outage. Measured with full verification, no `curl -k`: + # this URL answers /identity with 200 and TLS verify OK; the bare-IP form fails to connect. + # THE DASHED IP MUST MATCH /data/PLEX_L4_VIP ABOVE (192.168.2.24 -> 192-168-2-24); nothing + # enforces that. The hash is server identity, not a secret, and changes only if Plex reissues. + - op: replace + path: /data/PLEX_ADVERTISE_URL + value: "https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400" # The prod flat LAN. This is the 192.168.2.0/24 segment the SOHO ROUTER at 192.168.2.1 owns — # NOT pfSense's 192.168.1.0/24, which is a different network and not in the LAN-DNS path. # It scopes the ingress allow for PiHole's DNS listener to LAN clients only. diff --git a/decisions/2026-10-04-plex-advertise-plexdirect.md b/decisions/2026-10-04-plex-advertise-plexdirect.md new file mode 100644 index 0000000..20a12bf --- /dev/null +++ b/decisions/2026-10-04-plex-advertise-plexdirect.md @@ -0,0 +1,139 @@ +--- +date: 2026-10-04 +branch: fix/plex-advertise-plexdirect +title: Plex vanished from the plex.tv account — every advertised connection was uncompletable; advertise the plex.direct name instead of the bare VIP +--- + +## What changed + +Reported symptom: the server no longer appears on the operator's plex.tv account; `plex.home.keithjackson.dev` +shows "not authorized"; connecting straight to the VIP + port works. + +**Root cause, measured: `customConnections` held two hand-typed URLs and NEITHER could ever complete, +so Plex had nothing publishable.** Because `customConnections` OVERRIDES Plex's own connection +publishing, the server ended up advertising only dead endpoints. Live `Preferences.xml`, 2026-10-04: + + customConnections="https://plex.home.keithjackson.dev:32400/,https://192.168.2.24:32400/" + ManualPortMappingMode="1" ManualPortMappingPort= + +- `https://192.168.2.24:32400` — **TLS can never validate.** Plex serves 32400 with + `CN=*.130072f20c334d6c8f63d0dfef87309b.plex.direct` and **no IP SAN**, so a bare-IP https URL fails + for every client. Not theoretical: Plex's log carries a continuous storm of + `CERT: incomplete TLS handshake from ... sslv3 alert certificate unknown`, every few seconds, + from **real LAN clients** arriving through the new `plex-l4-tcp` VIP. +- `https://plex.home.keithjackson.dev:32400` — **nothing listens.** The shared Gateway serves that host + on 443; measured `http=000` (connect failed) against `192.168.2.20:32400`. + +Consequence chain, each link measured: no completable connection -> plex.tv could publish nothing +(`MyPlex: mapping state set to 'Mapped - Not Published'`, repeatedly) -> Plex fell back to **Plex Relay** +(`startRelay`, remote forward to 127.0.0.1:32401) -> **the server disappeared from the account**, and +clients reported "not authorized". + +The fix advertises Plex's own `..plex.direct` form, which is public DNS resolving +to the embedded address — so it points at the LAN VIP *and* matches the wildcard cert. + +- **`catalog/plex.yaml`:** `ADVERTISE_IP: ${PLEX_ADVERTISE_URL}` added, replacing the long-standing + DEFERRAL note. The deferral had weighed only the L7 Gateway URL; the operator chose the native L4 path + on 2026-10-03, so the open question was never "Gateway or not" but "which URL has a valid cert". +- **`clusters/base/cluster-vars.yaml`:** new `PLEX_ADVERTISE_URL` knob (+ substitution-inventory comment + updated to name `plex: PLEX_ADVERTISE_URL`). The base default is an explicit **non-functional + placeholder** — dev is decommissioned and has no claimed server, and + `StrictPostBuildSubstitutions` makes an undefined var a hard failure, so the default only has to exist + and be correctly shaped. +- **`clusters/prod/kustomization.yaml`:** prod value + `https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400`. +- **`apps/wave-0/plex.yaml`:** re-rendered with the canonical command + (`python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation`). Two added + lines, nothing else. + +**Why a whole-URL cluster-var and not something built from `${PLEX_L4_VIP}`:** postBuild substitution is +plain envsubst and cannot turn `192.168.2.24` into `192-168-2-24`, and `` is the server's +certificate identity, derivable from nothing in this tree. The dashed IP and `${PLEX_L4_VIP}` are two +halves of one address and **nothing checks that they agree** — noted in both files. + +**SENSITIVE PATHS — this PR touches `clusters/**` (two files), so `ci/approval-policy.sh` will BLOCK it.** +It needs the operator's `/approve-override ` posted as a PR **review** of type "Comment". +Deliberately not posted here: CLAUDE.md §1 forbids an agent clearing this on its own PR, and the agent +account holds `owner`, so the prohibition is a rule rather than a limit. + +## What I verified + +All measurements against live prod, 2026-10-04, read-only. Secrets handled per CLAUDE.md §6: preference +attributes were read by name-scoped grep, never by dumping `Preferences.xml` (it holds `PlexOnlineToken`), +and the token was reported only as ``. + +- **THE FIX ITSELF, with full TLS verification (no `curl -k`)** — the one measurement the whole change + rests on: + - `https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400/identity` -> **`http=200`, TLS verify OK** + - `https://192.168.2.24:32400/identity` -> **`http=000`, cert validation FAILED** + - that name resolves to `192.168.2.24` (`getent hosts`), confirming the dashed-IP mechanism works from + inside this network and is not blocked by DNS-rebinding protection on the resolver path tested. +- **The server is still claimed — this was never a lost claim.** `/identity` -> `claimed="1"` with the real + `machineIdentifier`; `PlexOnlineToken` present, `PlexOnlineUsername="jakman85"`. +- **The certificate, read off the live listener:** exactly one SAN, + `DNS:*.130072f20c334d6c8f63d0dfef87309b.plex.direct`. No IP SAN. This is why the bare-IP URL fails. +- **The publish payload Plex actually sent to plex.tv**, which is what proves the connection list was the + problem rather than a guess: `PUT https://servers.plex.tv/devices/?Connection[][uri]=https://plex.home.keithjackson.dev:32400&Connection[][uri]=https://192.168.2.24:32400&Connection[][uri]=http://136.51.115.252:32400&Connection[][uri]=http://10.0.2.136:32400&httpsEnabled=1...` + — all four unreachable from a LAN client or from plex.tv. +- **`Mapped - Not Published`** at every reachability check across the whole log, and an active **Plex Relay** + session — the documented fallback when nothing is directly publishable. +- **The `certificate unknown` TLS failures come from real client traffic, not a probe.** The source + `10.0.1.78` resolves via `cilium-dbg bpf ipcache get` to **identity 1 = `reserved:host`**, i.e. LAN + clients SNAT'd through `plex-l4-tcp` (`externalTrafficPolicy: Cluster`) on prod-3. +- **The Gateway is NOT broken, and my first reading of it was wrong.** A 403 `Access denied` from + `server: envoy` on the Gateway path turned out to be an artifact of probing from inside the plex pod: + **all 16** Gateway hostnames return 403 from there, because Cilium's Gateway preserves the original + source identity and enforces the destination's ingress policy. Plex's own log shows a concurrent, + **working, signed-in** browser session through the Gateway returning 200s. The HTTPRoute is + `Accepted=True`/`ResolvedRefs=True` with no filters, and it is the only route claiming that hostname. +- **Cilium is not denying the Gateway->Plex hop:** `cilium-dbg bpf policy get 3796` shows + `Allow Ingress reserved:ingress 32400/TCP`, PROXY PORT `NONE`, with 141k packets counted. No L7/HTTP + rules and no CCNPs exist anywhere in the cluster. +- **In-cluster consumers were never affected:** Seerr's recently-added scans and watchlist syncs complete + cleanly with no auth errors; there are **zero** HTTP 401s in Plex's log (the five ` 401 ` hits are + transcoder segment numbers). +- **`ADVERTISE_IP`'s mechanism, in the running image rather than from docs:** + `/etc/cont-init.d/40-plex-first-run` line 127 does `setPref "customConnections" "${ADVERTISE_IP}"`, + outside the claim guard, and `/.firstRunComplete` (lines 9/139) is on the container overlay, not the + `/config` PVC — so it re-applies on every pod start. Editing the env rolls the Deployment, which applies it. +- **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with + `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all + **pass**. The rc 3 is the two known environmental skips — `gitea-settings` (no local + `SETTINGS_API_TOKEN`) and `conformance` L2/L3 (`CLUSTER_AVAILABLE` unset). **Not green. Reported as rc 3.** +- **`render-golden` passing is the specific proof** that the catalog edit and the committed golden agree; + the rendered diff is exactly two lines and no other service's manifest moved. + +## What I did NOT verify + +- **THE OUTCOME. Nothing here has been observed to put the server back on the plex.tv account.** The + change is unreconciled: `ADVERTISE_IP` is absent from the live Deployment, the live + `customConnections` still holds the two broken URLs, and Plex has not been restarted. What is proven is + that the *new URL answers 200 with a valid certificate* and that *every currently advertised URL cannot + complete* — a necessary-and-plausibly-sufficient argument, not a demonstrated fix. +- **Whether plex.tv will publish and list the server once this lands.** plex.tv's decision is made + off-network and cannot be measured from here. A LAN-only connection may still leave the account showing + the server as remotely unreachable even when local clients work. +- **REMOTE (off-LAN) ACCESS IS NOT FIXED BY THIS, and may still be fully broken.** The advertised URL + resolves to an RFC1918 address, so it serves LAN clients only. The off-repo WAN tcp/32400 port-forward + (`runbooks/plex-remote-access.md`) is still **unverified** — `http://136.51.115.252:32400` was + unreachable from inside the LAN, but 2026-10-03 established there is no NAT hairpin here, so that is + **inconclusive**, not proof of absence. This repo has no off-LAN vantage point. +- **`ManualPortMappingMode="1"` with `ManualPortMappingPort` ABSENT is left as-is.** Manual public-port + mapping is enabled with no port set — a half-configured state I did not change and whose exact effect on + publishing I did not isolate. There is **no GitOps path** to it (2026-10-03: the image supports no + `PLEX_PREFERENCE*` of any kind), so it is an operator UI step. +- **Whether the operator intended the hand-typed URLs** to override the 2026-10-03 native-L4 design, or + typed them while debugging. I did not alter live preferences; the env var will overwrite + `customConnections` on the next pod start, which is a deliberate transfer of ownership from the web form + to git and will **silently discard** anything typed there since. +- **The `plex.direct` hash's stability.** Treated as server identity and read once from the live cert. I + did not establish what reissues it, so this URL could go stale with no alert; nothing in CI or the + conformance suite checks that the dashed IP still matches `${PLEX_L4_VIP}` or that the hash is current. +- **Resolver behaviour for the whole LAN.** `plex.direct` resolved correctly from a *pod* (cluster DNS + path). LAN clients resolve via PiHole, and dnsmasq's `stop-dns-rebind` would reject a public answer + carrying a private address. I did **not** test resolution from a LAN client, and this is the most likely + way the fix still fails. +- **The CI approval-policy outcome.** The local stage reported green, but that is an offline N/A, not a + verdict: this PR touches `clusters/**`, so CI will block pending the operator's override. +- **Why PMS restarted.** The process came up at 05:26 UTC while the pod is 2d2h old with 0 restarts, so + something restarted Plex inside the container. Unexplained and not investigated. -- 2.54.0