home-network-k3s-impl/decisions/2026-10-04-plex-advertise-plexdirect.md

date, branch, title

date	branch	title
2026-10-04	fix/plex-advertise-plexdirect	Plex vanished from the plex.tv account — every advertised connection was uncompletable; advertise the plex.direct name instead of the bare VIP

What changed

Reported symptom: the server no longer appears on the operator's plex.tv account; plex.home.keithjackson.dev shows "not authorized"; connecting straight to the VIP + port works.

Root cause, measured: customConnections held two hand-typed URLs and NEITHER could ever complete, so Plex had nothing publishable. Because customConnections OVERRIDES Plex's own connection publishing, the server ended up advertising only dead endpoints. Live Preferences.xml, 2026-10-04:

customConnections="https://plex.home.keithjackson.dev:32400/,https://192.168.2.24:32400/"
ManualPortMappingMode="1"   ManualPortMappingPort=<ABSENT>

• https://192.168.2.24:32400 — TLS can never validate. Plex serves 32400 with CN=*.130072f20c334d6c8f63d0dfef87309b.plex.direct and no IP SAN, so a bare-IP https URL fails for every client. Not theoretical: Plex's log carries a continuous storm of CERT: incomplete TLS handshake from <node> ... sslv3 alert certificate unknown, every few seconds, from real LAN clients arriving through the new plex-l4-tcp VIP.
• https://plex.home.keithjackson.dev:32400 — nothing listens. The shared Gateway serves that host on 443; measured http=000 (connect failed) against 192.168.2.20:32400.

Consequence chain, each link measured: no completable connection -> plex.tv could publish nothing (MyPlex: mapping state set to 'Mapped - Not Published', repeatedly) -> Plex fell back to Plex Relay (startRelay, remote forward to 127.0.0.1:32401) -> the server disappeared from the account, and clients reported "not authorized".

The fix advertises Plex's own <dashed-ip>.<serverhash>.plex.direct form, which is public DNS resolving to the embedded address — so it points at the LAN VIP and matches the wildcard cert.

• catalog/plex.yaml: ADVERTISE_IP: ${PLEX_ADVERTISE_URL} added, replacing the long-standing DEFERRAL note. The deferral had weighed only the L7 Gateway URL; the operator chose the native L4 path on 2026-10-03, so the open question was never "Gateway or not" but "which URL has a valid cert".
• clusters/base/cluster-vars.yaml: new PLEX_ADVERTISE_URL knob (+ substitution-inventory comment updated to name plex: PLEX_ADVERTISE_URL). The base default is an explicit non-functional placeholder — dev is decommissioned and has no claimed server, and StrictPostBuildSubstitutions makes an undefined var a hard failure, so the default only has to exist and be correctly shaped.
• clusters/prod/kustomization.yaml: prod value https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400.
• apps/wave-0/plex.yaml: re-rendered with the canonical command (python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation). Two added lines, nothing else.

Why a whole-URL cluster-var and not something built from ${PLEX_L4_VIP}: postBuild substitution is plain envsubst and cannot turn 192.168.2.24 into 192-168-2-24, and <serverhash> is the server's certificate identity, derivable from nothing in this tree. The dashed IP and ${PLEX_L4_VIP} are two halves of one address and nothing checks that they agree — noted in both files.

SENSITIVE PATHS — this PR touches clusters/** (two files), so ci/approval-policy.sh will BLOCK it. It needs the operator's /approve-override <reason> posted as a PR review of type "Comment". Deliberately not posted here: CLAUDE.md §1 forbids an agent clearing this on its own PR, and the agent account holds owner, so the prohibition is a rule rather than a limit.

What I verified

All measurements against live prod, 2026-10-04, read-only. Secrets handled per CLAUDE.md §6: preference attributes were read by name-scoped grep, never by dumping Preferences.xml (it holds PlexOnlineToken), and the token was reported only as <PRESENT,len=22>.

• THE FIX ITSELF, with full TLS verification (no curl -k) — the one measurement the whole change rests on:
  • https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400/identity -> http=200, TLS verify OK
  • https://192.168.2.24:32400/identity -> http=000, cert validation FAILED
  • that name resolves to 192.168.2.24 (getent hosts), confirming the dashed-IP mechanism works from inside this network and is not blocked by DNS-rebinding protection on the resolver path tested.
• The server is still claimed — this was never a lost claim. /identity -> claimed="1" with the real machineIdentifier; PlexOnlineToken present, PlexOnlineUsername="jakman85".
• The certificate, read off the live listener: exactly one SAN, DNS:*.130072f20c334d6c8f63d0dfef87309b.plex.direct. No IP SAN. This is why the bare-IP URL fails.
• The publish payload Plex actually sent to plex.tv, which is what proves the connection list was the problem rather than a guess: PUT https://servers.plex.tv/devices/<id>?Connection[][uri]=https://plex.home.keithjackson.dev:32400&Connection[][uri]=https://192.168.2.24:32400&Connection[][uri]=http://136.51.115.252:32400&Connection[][uri]=http://10.0.2.136:32400&httpsEnabled=1... — all four unreachable from a LAN client or from plex.tv.
• Mapped - Not Published at every reachability check across the whole log, and an active Plex Relay session — the documented fallback when nothing is directly publishable.
• The certificate unknown TLS failures come from real client traffic, not a probe. The source 10.0.1.78 resolves via cilium-dbg bpf ipcache get to identity 1 = reserved:host, i.e. LAN clients SNAT'd through plex-l4-tcp (externalTrafficPolicy: Cluster) on prod-3.
• The Gateway is NOT broken, and my first reading of it was wrong. A 403 Access denied from server: envoy on the Gateway path turned out to be an artifact of probing from inside the plex pod: all 16 Gateway hostnames return 403 from there, because Cilium's Gateway preserves the original source identity and enforces the destination's ingress policy. Plex's own log shows a concurrent, working, signed-in browser session through the Gateway returning 200s. The HTTPRoute is Accepted=True/ResolvedRefs=True with no filters, and it is the only route claiming that hostname.
• Cilium is not denying the Gateway->Plex hop: cilium-dbg bpf policy get 3796 shows Allow Ingress reserved:ingress 32400/TCP, PROXY PORT NONE, with 141k packets counted. No L7/HTTP rules and no CCNPs exist anywhere in the cluster.
• In-cluster consumers were never affected: Seerr's recently-added scans and watchlist syncs complete cleanly with no auth errors; there are zero HTTP 401s in Plex's log (the five 401 hits are transcoder segment numbers).
• ADVERTISE_IP's mechanism, in the running image rather than from docs: /etc/cont-init.d/40-plex-first-run line 127 does setPref "customConnections" "${ADVERTISE_IP}", outside the claim guard, and /.firstRunComplete (lines 9/139) is on the container overlay, not the /config PVC — so it re-applies on every pod start. Editing the env rolls the Deployment, which applies it.
• Toolbox gate (./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'): rc 3, with spec-immutability, decisions-log, lint, l0, render-golden, l1 and approval-policy all pass. The rc 3 is the two known environmental skips — gitea-settings (no local SETTINGS_API_TOKEN) and conformance L2/L3 (CLUSTER_AVAILABLE unset). Not green. Reported as rc 3.
• render-golden passing is the specific proof that the catalog edit and the committed golden agree; the rendered diff is exactly two lines and no other service's manifest moved.

What I did NOT verify

• THE OUTCOME. Nothing here has been observed to put the server back on the plex.tv account. The change is unreconciled: ADVERTISE_IP is absent from the live Deployment, the live customConnections still holds the two broken URLs, and Plex has not been restarted. What is proven is that the new URL answers 200 with a valid certificate and that every currently advertised URL cannot complete — a necessary-and-plausibly-sufficient argument, not a demonstrated fix.
• Whether plex.tv will publish and list the server once this lands. plex.tv's decision is made off-network and cannot be measured from here. A LAN-only connection may still leave the account showing the server as remotely unreachable even when local clients work.
• REMOTE (off-LAN) ACCESS IS NOT FIXED BY THIS, and may still be fully broken. The advertised URL resolves to an RFC1918 address, so it serves LAN clients only. The off-repo WAN tcp/32400 port-forward (runbooks/plex-remote-access.md) is still unverified — http://136.51.115.252:32400 was unreachable from inside the LAN, but 2026-10-03 established there is no NAT hairpin here, so that is inconclusive, not proof of absence. This repo has no off-LAN vantage point.
• ManualPortMappingMode="1" with ManualPortMappingPort ABSENT is left as-is. Manual public-port mapping is enabled with no port set — a half-configured state I did not change and whose exact effect on publishing I did not isolate. There is no GitOps path to it (2026-10-03: the image supports no PLEX_PREFERENCE* of any kind), so it is an operator UI step.
• Whether the operator intended the hand-typed URLs to override the 2026-10-03 native-L4 design, or typed them while debugging. I did not alter live preferences; the env var will overwrite customConnections on the next pod start, which is a deliberate transfer of ownership from the web form to git and will silently discard anything typed there since.
• The plex.direct hash's stability. Treated as server identity and read once from the live cert. I did not establish what reissues it, so this URL could go stale with no alert; nothing in CI or the conformance suite checks that the dashed IP still matches ${PLEX_L4_VIP} or that the hash is current.
• Resolver behaviour for the whole LAN. plex.direct resolved correctly from a pod (cluster DNS path). LAN clients resolve via PiHole, and dnsmasq's stop-dns-rebind would reject a public answer carrying a private address. I did not test resolution from a LAN client, and this is the most likely way the fix still fails.
• The CI approval-policy outcome. The local stage reported green, but that is an offline N/A, not a verdict: this PR touches clusters/**, so CI will block pending the operator's override.
• Why PMS restarted. The process came up at 05:26 UTC while the pod is 2d2h old with 0 restarts, so something restarted Plex inside the container. Unexplained and not investigated.