home-network-k3s-impl/decisions/2026-10-04-plex-advertise-plexdirect.md

---
date: 2026-10-04
branch: fix/plex-advertise-plexdirect
title: Plex vanished from the plex.tv account — every advertised connection was uncompletable; advertise the plex.direct name instead of the bare VIP
---

## What changed

Reported symptom: the server no longer appears on the operator's plex.tv account; `plex.home.keithjackson.dev`
shows "not authorized"; connecting straight to the VIP + port works.

**Root cause, measured: `customConnections` held two hand-typed URLs and NEITHER could ever complete,
so Plex had nothing publishable.** Because `customConnections` OVERRIDES Plex's own connection
publishing, the server ended up advertising only dead endpoints. Live `Preferences.xml`, 2026-10-04:

    customConnections="https://plex.home.keithjackson.dev:32400/,https://192.168.2.24:32400/"
    ManualPortMappingMode="1"   ManualPortMappingPort=<ABSENT>

- `https://192.168.2.24:32400` — **TLS can never validate.** Plex serves 32400 with
  `CN=*.130072f20c334d6c8f63d0dfef87309b.plex.direct` and **no IP SAN**, so a bare-IP https URL fails
  for every client. Not theoretical: Plex's log carries a continuous storm of
  `CERT: incomplete TLS handshake from <node> ... sslv3 alert certificate unknown`, every few seconds,
  from **real LAN clients** arriving through the new `plex-l4-tcp` VIP.
- `https://plex.home.keithjackson.dev:32400` — **nothing listens.** The shared Gateway serves that host
  on 443; measured `http=000` (connect failed) against `192.168.2.20:32400`.

Consequence chain, each link measured: no completable connection -> plex.tv could publish nothing
(`MyPlex: mapping state set to 'Mapped - Not Published'`, repeatedly) -> Plex fell back to **Plex Relay**
(`startRelay`, remote forward to 127.0.0.1:32401) -> **the server disappeared from the account**, and
clients reported "not authorized".

The fix advertises Plex's own `<dashed-ip>.<serverhash>.plex.direct` form, which is public DNS resolving
to the embedded address — so it points at the LAN VIP *and* matches the wildcard cert.

- **`catalog/plex.yaml`:** `ADVERTISE_IP: ${PLEX_ADVERTISE_URL}` added, replacing the long-standing
  DEFERRAL note. The deferral had weighed only the L7 Gateway URL; the operator chose the native L4 path
  on 2026-10-03, so the open question was never "Gateway or not" but "which URL has a valid cert".
- **`clusters/base/cluster-vars.yaml`:** new `PLEX_ADVERTISE_URL` knob (+ substitution-inventory comment
  updated to name `plex: PLEX_ADVERTISE_URL`). The base default is an explicit **non-functional
  placeholder** — dev is decommissioned and has no claimed server, and
  `StrictPostBuildSubstitutions` makes an undefined var a hard failure, so the default only has to exist
  and be correctly shaped.
- **`clusters/prod/kustomization.yaml`:** prod value
  `https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400`.
- **`apps/wave-0/plex.yaml`:** re-rendered with the canonical command
  (`python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation`). Two added
  lines, nothing else.

**Why a whole-URL cluster-var and not something built from `${PLEX_L4_VIP}`:** postBuild substitution is
plain envsubst and cannot turn `192.168.2.24` into `192-168-2-24`, and `<serverhash>` is the server's
certificate identity, derivable from nothing in this tree. The dashed IP and `${PLEX_L4_VIP}` are two
halves of one address and **nothing checks that they agree** — noted in both files.

**SENSITIVE PATHS — this PR touches `clusters/**` (two files), so `ci/approval-policy.sh` will BLOCK it.**
It needs the operator's `/approve-override <reason>` posted as a PR **review** of type "Comment".
Deliberately not posted here: CLAUDE.md §1 forbids an agent clearing this on its own PR, and the agent
account holds `owner`, so the prohibition is a rule rather than a limit.

## What I verified

All measurements against live prod, 2026-10-04, read-only. Secrets handled per CLAUDE.md §6: preference
attributes were read by name-scoped grep, never by dumping `Preferences.xml` (it holds `PlexOnlineToken`),
and the token was reported only as `<PRESENT,len=22>`.

- **THE FIX ITSELF, with full TLS verification (no `curl -k`)** — the one measurement the whole change
  rests on:
  - `https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400/identity` -> **`http=200`, TLS verify OK**
  - `https://192.168.2.24:32400/identity` -> **`http=000`, cert validation FAILED**
  - that name resolves to `192.168.2.24` (`getent hosts`), confirming the dashed-IP mechanism works from
    inside this network and is not blocked by DNS-rebinding protection on the resolver path tested.
- **The server is still claimed — this was never a lost claim.** `/identity` -> `claimed="1"` with the real
  `machineIdentifier`; `PlexOnlineToken` present, `PlexOnlineUsername="jakman85"`.
- **The certificate, read off the live listener:** exactly one SAN,
  `DNS:*.130072f20c334d6c8f63d0dfef87309b.plex.direct`. No IP SAN. This is why the bare-IP URL fails.
- **The publish payload Plex actually sent to plex.tv**, which is what proves the connection list was the
  problem rather than a guess: `PUT https://servers.plex.tv/devices/<id>?Connection[][uri]=https://plex.home.keithjackson.dev:32400&Connection[][uri]=https://192.168.2.24:32400&Connection[][uri]=http://136.51.115.252:32400&Connection[][uri]=http://10.0.2.136:32400&httpsEnabled=1...`
  — all four unreachable from a LAN client or from plex.tv.
- **`Mapped - Not Published`** at every reachability check across the whole log, and an active **Plex Relay**
  session — the documented fallback when nothing is directly publishable.
- **The `certificate unknown` TLS failures come from real client traffic, not a probe.** The source
  `10.0.1.78` resolves via `cilium-dbg bpf ipcache get` to **identity 1 = `reserved:host`**, i.e. LAN
  clients SNAT'd through `plex-l4-tcp` (`externalTrafficPolicy: Cluster`) on prod-3.
- **The Gateway is NOT broken, and my first reading of it was wrong.** A 403 `Access denied` from
  `server: envoy` on the Gateway path turned out to be an artifact of probing from inside the plex pod:
  **all 16** Gateway hostnames return 403 from there, because Cilium's Gateway preserves the original
  source identity and enforces the destination's ingress policy. Plex's own log shows a concurrent,
  **working, signed-in** browser session through the Gateway returning 200s. The HTTPRoute is
  `Accepted=True`/`ResolvedRefs=True` with no filters, and it is the only route claiming that hostname.
- **Cilium is not denying the Gateway->Plex hop:** `cilium-dbg bpf policy get 3796` shows
  `Allow Ingress reserved:ingress 32400/TCP`, PROXY PORT `NONE`, with 141k packets counted. No L7/HTTP
  rules and no CCNPs exist anywhere in the cluster.
- **In-cluster consumers were never affected:** Seerr's recently-added scans and watchlist syncs complete
  cleanly with no auth errors; there are **zero** HTTP 401s in Plex's log (the five ` 401 ` hits are
  transcoder segment numbers).
- **`ADVERTISE_IP`'s mechanism, in the running image rather than from docs:**
  `/etc/cont-init.d/40-plex-first-run` line 127 does `setPref "customConnections" "${ADVERTISE_IP}"`,
  outside the claim guard, and `/.firstRunComplete` (lines 9/139) is on the container overlay, not the
  `/config` PVC — so it re-applies on every pod start. Editing the env rolls the Deployment, which applies it.
- **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with
  `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all
  **pass**. The rc 3 is the two known environmental skips — `gitea-settings` (no local
  `SETTINGS_API_TOKEN`) and `conformance` L2/L3 (`CLUSTER_AVAILABLE` unset). **Not green. Reported as rc 3.**
- **`render-golden` passing is the specific proof** that the catalog edit and the committed golden agree;
  the rendered diff is exactly two lines and no other service's manifest moved.

## What I did NOT verify

- **THE OUTCOME. Nothing here has been observed to put the server back on the plex.tv account.** The
  change is unreconciled: `ADVERTISE_IP` is absent from the live Deployment, the live
  `customConnections` still holds the two broken URLs, and Plex has not been restarted. What is proven is
  that the *new URL answers 200 with a valid certificate* and that *every currently advertised URL cannot
  complete* — a necessary-and-plausibly-sufficient argument, not a demonstrated fix.
- **Whether plex.tv will publish and list the server once this lands.** plex.tv's decision is made
  off-network and cannot be measured from here. A LAN-only connection may still leave the account showing
  the server as remotely unreachable even when local clients work.
- **REMOTE (off-LAN) ACCESS IS NOT FIXED BY THIS, and may still be fully broken.** The advertised URL
  resolves to an RFC1918 address, so it serves LAN clients only. The off-repo WAN tcp/32400 port-forward
  (`runbooks/plex-remote-access.md`) is still **unverified** — `http://136.51.115.252:32400` was
  unreachable from inside the LAN, but 2026-10-03 established there is no NAT hairpin here, so that is
  **inconclusive**, not proof of absence. This repo has no off-LAN vantage point.
- **`ManualPortMappingMode="1"` with `ManualPortMappingPort` ABSENT is left as-is.** Manual public-port
  mapping is enabled with no port set — a half-configured state I did not change and whose exact effect on
  publishing I did not isolate. There is **no GitOps path** to it (2026-10-03: the image supports no
  `PLEX_PREFERENCE*` of any kind), so it is an operator UI step.
- **Whether the operator intended the hand-typed URLs** to override the 2026-10-03 native-L4 design, or
  typed them while debugging. I did not alter live preferences; the env var will overwrite
  `customConnections` on the next pod start, which is a deliberate transfer of ownership from the web form
  to git and will **silently discard** anything typed there since.
- **The `plex.direct` hash's stability.** Treated as server identity and read once from the live cert. I
  did not establish what reissues it, so this URL could go stale with no alert; nothing in CI or the
  conformance suite checks that the dashed IP still matches `${PLEX_L4_VIP}` or that the hash is current.
- **Resolver behaviour for the whole LAN.** `plex.direct` resolved correctly from a *pod* (cluster DNS
  path). LAN clients resolve via PiHole, and dnsmasq's `stop-dns-rebind` would reject a public answer
  carrying a private address. I did **not** test resolution from a LAN client, and this is the most likely
  way the fix still fails.
- **The CI approval-policy outcome.** The local stage reported green, but that is an offline N/A, not a
  verdict: this PR touches `clusters/**`, so CI will block pending the operator's override.
- **Why PMS restarted.** The process came up at 05:26 UTC while the pod is 2d2h old with 0 restarts, so
  something restarted Plex inside the container. Unexplained and not investigated.