fix(plex): advertise the plex.direct name, not the bare VIP (server vanished from the plex.tv account)
customConnections held two hand-typed URLs and neither could ever complete, so Plex had nothing publishable and the server disappeared from the account: - https://192.168.2.24:32400 -- the server's cert is *.130072f20c334d6c8f63d0dfef87309b.plex.direct with NO IP SAN, so bare-IP https fails TLS for every client. Measured: a continuous storm of "CERT: incomplete TLS handshake ... certificate unknown" from real LAN clients arriving through the plex-l4-tcp VIP. - https://plex.home.keithjackson.dev:32400 -- nothing listens; the Gateway serves that host on 443 (measured http=000 against 192.168.2.20:32400). With no completable connection plex.tv published nothing ("Mapped - Not Published"), Plex fell back to Relay, and clients reported "not authorized". Fix: advertise <dashed-ip>.<serverhash>.plex.direct -- public DNS that resolves to the embedded address, so it points at the LAN VIP and matches the wildcard cert. Verified with full TLS validation (no curl -k): 200 + verify OK, where the bare-IP form fails to connect. Declared as ADVERTISE_IP so git owns the value instead of a web form: the image's 40-plex-first-run writes it into customConnections on every pod start, outside the claim guard. Does NOT fix off-LAN access -- the advertised address is RFC1918 and the WAN tcp/32400 forward remains unverified. ManualPortMappingMode=1 with an absent port is left as-is (no GitOps path exists for it). Touches clusters/** -> sensitive paths -> needs the operator's /approve-override, deliberately not self-posted (CLAUDE.md 1). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
c95b1c7121
commit
1f4552d409
5 files changed
+226
-9
No files matched your search
@@ -0,0 +1,139 @@
|
||||
---
|
||||
date: 2026-10-04
|
||||
branch: fix/plex-advertise-plexdirect
|
||||
title: Plex vanished from the plex.tv account — every advertised connection was uncompletable; advertise the plex.direct name instead of the bare VIP
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Reported symptom: the server no longer appears on the operator's plex.tv account; `plex.home.keithjackson.dev`
|
||||
shows "not authorized"; connecting straight to the VIP + port works.
|
||||
|
||||
**Root cause, measured: `customConnections` held two hand-typed URLs and NEITHER could ever complete,
|
||||
so Plex had nothing publishable.** Because `customConnections` OVERRIDES Plex's own connection
|
||||
publishing, the server ended up advertising only dead endpoints. Live `Preferences.xml`, 2026-10-04:
|
||||
|
||||
customConnections="https://plex.home.keithjackson.dev:32400/,https://192.168.2.24:32400/"
|
||||
ManualPortMappingMode="1" ManualPortMappingPort=<ABSENT>
|
||||
|
||||
- `https://192.168.2.24:32400` — **TLS can never validate.** Plex serves 32400 with
|
||||
`CN=*.130072f20c334d6c8f63d0dfef87309b.plex.direct` and **no IP SAN**, so a bare-IP https URL fails
|
||||
for every client. Not theoretical: Plex's log carries a continuous storm of
|
||||
`CERT: incomplete TLS handshake from <node> ... sslv3 alert certificate unknown`, every few seconds,
|
||||
from **real LAN clients** arriving through the new `plex-l4-tcp` VIP.
|
||||
- `https://plex.home.keithjackson.dev:32400` — **nothing listens.** The shared Gateway serves that host
|
||||
on 443; measured `http=000` (connect failed) against `192.168.2.20:32400`.
|
||||
|
||||
Consequence chain, each link measured: no completable connection -> plex.tv could publish nothing
|
||||
(`MyPlex: mapping state set to 'Mapped - Not Published'`, repeatedly) -> Plex fell back to **Plex Relay**
|
||||
(`startRelay`, remote forward to 127.0.0.1:32401) -> **the server disappeared from the account**, and
|
||||
clients reported "not authorized".
|
||||
|
||||
The fix advertises Plex's own `<dashed-ip>.<serverhash>.plex.direct` form, which is public DNS resolving
|
||||
to the embedded address — so it points at the LAN VIP *and* matches the wildcard cert.
|
||||
|
||||
- **`catalog/plex.yaml`:** `ADVERTISE_IP: ${PLEX_ADVERTISE_URL}` added, replacing the long-standing
|
||||
DEFERRAL note. The deferral had weighed only the L7 Gateway URL; the operator chose the native L4 path
|
||||
on 2026-10-03, so the open question was never "Gateway or not" but "which URL has a valid cert".
|
||||
- **`clusters/base/cluster-vars.yaml`:** new `PLEX_ADVERTISE_URL` knob (+ substitution-inventory comment
|
||||
updated to name `plex: PLEX_ADVERTISE_URL`). The base default is an explicit **non-functional
|
||||
placeholder** — dev is decommissioned and has no claimed server, and
|
||||
`StrictPostBuildSubstitutions` makes an undefined var a hard failure, so the default only has to exist
|
||||
and be correctly shaped.
|
||||
- **`clusters/prod/kustomization.yaml`:** prod value
|
||||
`https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400`.
|
||||
- **`apps/wave-0/plex.yaml`:** re-rendered with the canonical command
|
||||
(`python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation`). Two added
|
||||
lines, nothing else.
|
||||
|
||||
**Why a whole-URL cluster-var and not something built from `${PLEX_L4_VIP}`:** postBuild substitution is
|
||||
plain envsubst and cannot turn `192.168.2.24` into `192-168-2-24`, and `<serverhash>` is the server's
|
||||
certificate identity, derivable from nothing in this tree. The dashed IP and `${PLEX_L4_VIP}` are two
|
||||
halves of one address and **nothing checks that they agree** — noted in both files.
|
||||
|
||||
**SENSITIVE PATHS — this PR touches `clusters/**` (two files), so `ci/approval-policy.sh` will BLOCK it.**
|
||||
It needs the operator's `/approve-override <reason>` posted as a PR **review** of type "Comment".
|
||||
Deliberately not posted here: CLAUDE.md §1 forbids an agent clearing this on its own PR, and the agent
|
||||
account holds `owner`, so the prohibition is a rule rather than a limit.
|
||||
|
||||
## What I verified
|
||||
|
||||
All measurements against live prod, 2026-10-04, read-only. Secrets handled per CLAUDE.md §6: preference
|
||||
attributes were read by name-scoped grep, never by dumping `Preferences.xml` (it holds `PlexOnlineToken`),
|
||||
and the token was reported only as `<PRESENT,len=22>`.
|
||||
|
||||
- **THE FIX ITSELF, with full TLS verification (no `curl -k`)** — the one measurement the whole change
|
||||
rests on:
|
||||
- `https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400/identity` -> **`http=200`, TLS verify OK**
|
||||
- `https://192.168.2.24:32400/identity` -> **`http=000`, cert validation FAILED**
|
||||
- that name resolves to `192.168.2.24` (`getent hosts`), confirming the dashed-IP mechanism works from
|
||||
inside this network and is not blocked by DNS-rebinding protection on the resolver path tested.
|
||||
- **The server is still claimed — this was never a lost claim.** `/identity` -> `claimed="1"` with the real
|
||||
`machineIdentifier`; `PlexOnlineToken` present, `PlexOnlineUsername="jakman85"`.
|
||||
- **The certificate, read off the live listener:** exactly one SAN,
|
||||
`DNS:*.130072f20c334d6c8f63d0dfef87309b.plex.direct`. No IP SAN. This is why the bare-IP URL fails.
|
||||
- **The publish payload Plex actually sent to plex.tv**, which is what proves the connection list was the
|
||||
problem rather than a guess: `PUT https://servers.plex.tv/devices/<id>?Connection[][uri]=https://plex.home.keithjackson.dev:32400&Connection[][uri]=https://192.168.2.24:32400&Connection[][uri]=http://136.51.115.252:32400&Connection[][uri]=http://10.0.2.136:32400&httpsEnabled=1...`
|
||||
— all four unreachable from a LAN client or from plex.tv.
|
||||
- **`Mapped - Not Published`** at every reachability check across the whole log, and an active **Plex Relay**
|
||||
session — the documented fallback when nothing is directly publishable.
|
||||
- **The `certificate unknown` TLS failures come from real client traffic, not a probe.** The source
|
||||
`10.0.1.78` resolves via `cilium-dbg bpf ipcache get` to **identity 1 = `reserved:host`**, i.e. LAN
|
||||
clients SNAT'd through `plex-l4-tcp` (`externalTrafficPolicy: Cluster`) on prod-3.
|
||||
- **The Gateway is NOT broken, and my first reading of it was wrong.** A 403 `Access denied` from
|
||||
`server: envoy` on the Gateway path turned out to be an artifact of probing from inside the plex pod:
|
||||
**all 16** Gateway hostnames return 403 from there, because Cilium's Gateway preserves the original
|
||||
source identity and enforces the destination's ingress policy. Plex's own log shows a concurrent,
|
||||
**working, signed-in** browser session through the Gateway returning 200s. The HTTPRoute is
|
||||
`Accepted=True`/`ResolvedRefs=True` with no filters, and it is the only route claiming that hostname.
|
||||
- **Cilium is not denying the Gateway->Plex hop:** `cilium-dbg bpf policy get 3796` shows
|
||||
`Allow Ingress reserved:ingress 32400/TCP`, PROXY PORT `NONE`, with 141k packets counted. No L7/HTTP
|
||||
rules and no CCNPs exist anywhere in the cluster.
|
||||
- **In-cluster consumers were never affected:** Seerr's recently-added scans and watchlist syncs complete
|
||||
cleanly with no auth errors; there are **zero** HTTP 401s in Plex's log (the five ` 401 ` hits are
|
||||
transcoder segment numbers).
|
||||
- **`ADVERTISE_IP`'s mechanism, in the running image rather than from docs:**
|
||||
`/etc/cont-init.d/40-plex-first-run` line 127 does `setPref "customConnections" "${ADVERTISE_IP}"`,
|
||||
outside the claim guard, and `/.firstRunComplete` (lines 9/139) is on the container overlay, not the
|
||||
`/config` PVC — so it re-applies on every pod start. Editing the env rolls the Deployment, which applies it.
|
||||
- **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with
|
||||
`spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all
|
||||
**pass**. The rc 3 is the two known environmental skips — `gitea-settings` (no local
|
||||
`SETTINGS_API_TOKEN`) and `conformance` L2/L3 (`CLUSTER_AVAILABLE` unset). **Not green. Reported as rc 3.**
|
||||
- **`render-golden` passing is the specific proof** that the catalog edit and the committed golden agree;
|
||||
the rendered diff is exactly two lines and no other service's manifest moved.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **THE OUTCOME. Nothing here has been observed to put the server back on the plex.tv account.** The
|
||||
change is unreconciled: `ADVERTISE_IP` is absent from the live Deployment, the live
|
||||
`customConnections` still holds the two broken URLs, and Plex has not been restarted. What is proven is
|
||||
that the *new URL answers 200 with a valid certificate* and that *every currently advertised URL cannot
|
||||
complete* — a necessary-and-plausibly-sufficient argument, not a demonstrated fix.
|
||||
- **Whether plex.tv will publish and list the server once this lands.** plex.tv's decision is made
|
||||
off-network and cannot be measured from here. A LAN-only connection may still leave the account showing
|
||||
the server as remotely unreachable even when local clients work.
|
||||
- **REMOTE (off-LAN) ACCESS IS NOT FIXED BY THIS, and may still be fully broken.** The advertised URL
|
||||
resolves to an RFC1918 address, so it serves LAN clients only. The off-repo WAN tcp/32400 port-forward
|
||||
(`runbooks/plex-remote-access.md`) is still **unverified** — `http://136.51.115.252:32400` was
|
||||
unreachable from inside the LAN, but 2026-10-03 established there is no NAT hairpin here, so that is
|
||||
**inconclusive**, not proof of absence. This repo has no off-LAN vantage point.
|
||||
- **`ManualPortMappingMode="1"` with `ManualPortMappingPort` ABSENT is left as-is.** Manual public-port
|
||||
mapping is enabled with no port set — a half-configured state I did not change and whose exact effect on
|
||||
publishing I did not isolate. There is **no GitOps path** to it (2026-10-03: the image supports no
|
||||
`PLEX_PREFERENCE*` of any kind), so it is an operator UI step.
|
||||
- **Whether the operator intended the hand-typed URLs** to override the 2026-10-03 native-L4 design, or
|
||||
typed them while debugging. I did not alter live preferences; the env var will overwrite
|
||||
`customConnections` on the next pod start, which is a deliberate transfer of ownership from the web form
|
||||
to git and will **silently discard** anything typed there since.
|
||||
- **The `plex.direct` hash's stability.** Treated as server identity and read once from the live cert. I
|
||||
did not establish what reissues it, so this URL could go stale with no alert; nothing in CI or the
|
||||
conformance suite checks that the dashed IP still matches `${PLEX_L4_VIP}` or that the hash is current.
|
||||
- **Resolver behaviour for the whole LAN.** `plex.direct` resolved correctly from a *pod* (cluster DNS
|
||||
path). LAN clients resolve via PiHole, and dnsmasq's `stop-dns-rebind` would reject a public answer
|
||||
carrying a private address. I did **not** test resolution from a LAN client, and this is the most likely
|
||||
way the fix still fails.
|
||||
- **The CI approval-policy outcome.** The local stage reported green, but that is an offline N/A, not a
|
||||
verdict: this PR touches `clusters/**`, so CI will block pending the operator's override.
|
||||
- **Why PMS restarted.** The process came up at 05:26 UTC while the pod is 2d2h old with 0 restarts, so
|
||||
something restarted Plex inside the container. Unexplained and not investigated.
|
||||
Reference in new issue
Block a user