Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

fix(plex): advertise the plex.direct name, not the bare VIP (server vanished from the plex.tv account) #47

Merged
jakman85 merged 1 commits from fix/plex-advertise-plexdirect into main 2026-10-04 07:59:14 +00:00
Conversation 1 Commits 1 Files Changed 5
+226 -9
5 changed files with 226 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

apps/wave-0/plex.yaml
+2
View File
@@ -137,6 +137,8 @@ spec:
secretKeyRef:
name: plex-claim
key: PLEX_CLAIM
- name: ADVERTISE_IP
value: ${PLEX_ADVERTISE_URL}
resources:
limits:
gpu.intel.com/i915: 1
catalog/plex.yaml
+48 -8
View File
@@ -72,14 +72,54 @@ env:
secretKeyRef:
name: plex-claim
key: PLEX_CLAIM
# ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex
# which URL to advertise to plex.tv for REMOTE clients. The only candidate is
# https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL
# Gateway VIP (conformance/tests/l2-behavioral/dns-internal-resolution), and both its public
# reachability and whether the TLS-terminating Cilium Gateway proxies Plex's 32400 API correctly for
# remote access are UNVERIFIED. Advertising a wrong/internal-only URL breaks remote access for ALL
# clients, so per "prefer safety over completeness" it is left unset. FOLLOW-UP: once remote
# reachability through the Gateway is confirmed, set ADVERTISE_IP: "https://plex.home.keithjackson.dev:443".
# ADVERTISE_IP ("Custom server access URLs") — NOW SET. This replaces a DEFERRAL note that stood
# here until 2026-10-04, and the deferral's premise has changed: it weighed only ONE candidate,
# `https://plex.home.keithjackson.dev` through the L7 Gateway. The operator chose the NATIVE L4
# path instead on 2026-10-03 (apps/wave-0/plex-remote.yaml) so streams go straight to
# ${PLEX_L4_VIP}:32400 and bypass the Envoy hop. What was still missing was a URL whose TLS
# CERTIFICATE validates — and getting that wrong is what took the service down.
#
# THE VALUE IS A `plex.direct` NAME, NOT THE RAW VIP, AND THAT IS THE WHOLE POINT. Plex serves
# 32400 with its own certificate; measured live 2026-10-04 it carries exactly one name:
# CN = *.<serverhash>.plex.direct (one SAN, no IP SANs at all)
# so `https://<ip>:32400` can NEVER validate. `<dashed-ip>.<serverhash>.plex.direct` is Plex's own
# mechanism for this: it is PUBLIC DNS that resolves to the embedded address, so the name points at
# the LAN VIP while the wildcard cert above matches it. Measured 2026-10-04 with FULL verification
# (no `curl -k`):
# https://192-168-2-24.<hash>.plex.direct:32400/identity -> 200, TLS verify OK
# https://192.168.2.24:32400/identity -> 000, cert validation FAILED
#
# THIS FIELD EXISTS BECAUSE AUTO-DISCOVERY CANNOT WORK HERE. Left empty, Plex publishes the
# addresses it can see itself — which from a pod is the overlay IP (`10.0.2.136`), useless to a LAN
# client. The VIP has to be advertised explicitly; there is no "just clear it" fix.
#
# WHAT WENT WRONG, recorded so it is not re-introduced by hand. customConnections OVERRIDES Plex's
# own publishing, and on 2026-10-04 it held two hand-typed URLs, both uncompletable:
# - `https://192.168.2.24:32400` -> cert cannot cover an IP. Measured: a storm of
# `CERT: incomplete TLS handshake ... sslv3 alert certificate unknown` in Plex's log every few
# seconds, from REAL LAN clients arriving through the VIP.
# - `https://plex.home.keithjackson.dev:32400` -> nothing listens. The Gateway serves this host on
# 443; measured `http=000` against 192.168.2.20:32400.
# With every advertised connection dead, plex.tv had nothing publishable, sat at
# `mapping state 'Mapped - Not Published'`, fell back to Plex Relay, and THE SERVER DISAPPEARED
# FROM THE ACCOUNT — clients reported "not authorized".
#
# DECLARING IT HERE rather than in the web UI is the durable half of the fix. The image's
# /etc/cont-init.d/40-plex-first-run runs `setPref "customConnections" "${ADVERTISE_IP}"` (line 127)
# OUTSIDE the PLEX_CLAIM/PlexOnlineToken guard, and its `/.firstRunComplete` marker sits on the
# container's OVERLAY fs, not the /config PVC — so this is re-applied on EVERY pod start and git,
# not a form field, owns the value. Editing this env rolls the Deployment, which is what applies it.
#
# TWO LIMITS, stated rather than discovered later:
# - THE HASH IS SERVER IDENTITY, which is why the whole URL is a cluster-var and not built from
# ${PLEX_L4_VIP}: envsubst cannot dash an IP, and the hash is not derivable from anything in
# this tree. It is not a secret (it ships in the cert and in the published connection list), but
# if Plex reissues under a new hash this URL goes stale and must be re-read from the live cert.
# - THIS RESTORES LAN CLIENTS AND THE ACCOUNT LISTING, NOT WAN. Remote clients still need the
# off-repo tcp/32400 port-forward (runbooks/plex-remote-access.md). Once that exists, the WAN
# twin of this URL can be appended comma-separated — customConnections takes a list.
- name: ADVERTISE_IP
value: ${PLEX_ADVERTISE_URL}
# HARDWARE TRANSCODING — the host iGPU's render device (schema 1.8.0 `hardware`; R-PLAT-02).
# INTENT ONLY, AND THE MECHANISM DELIBERATELY DOES NOT LIVE HERE. This says "this workload needs an
# Intel Quick Sync render device"; it does NOT name a device path and it does NOT name a resource.
clusters/base/cluster-vars.yaml
+27 -1
View File
@@ -101,7 +101,7 @@ data:
# tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
# S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
# GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers;
# plex-remote: PLEX_L4_VIP),
# plex-remote: PLEX_L4_VIP; plex: PLEX_ADVERTISE_URL),
# velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
# VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
# UNSUBSTITUTED path would ship the literal string into the cluster.
@@ -190,6 +190,32 @@ data:
# prod -> 192.168.2.24 (next free after pihole-dns .23 in prod's .20-.30 pool; see clusters/prod)
PLEX_L4_VIP: "192.168.1.57" # Plex: tcp/32400 remote access (NOT the web UI, which is L7)
# PLEX_ADVERTISE_URL — the connection URL Plex publishes to plex.tv ("Custom server access URLs",
# the ADVERTISE_IP env in catalog/plex.yaml, which the image writes into the `customConnections`
# preference on every pod start). Read catalog/plex.yaml's ADVERTISE_IP block for the full why;
# the short version is the only thing that matters when editing this line:
#
# IT MUST BE A `<dashed-ip>.<serverhash>.plex.direct` NAME, NEVER A BARE IP. Plex serves 32400
# with a `*.<serverhash>.plex.direct` wildcard certificate and NO IP SAN, so `https://<ip>:32400`
# fails TLS validation for every client. That exact mistake is what broke this service on
# 2026-10-04: with no completable connection, plex.tv published nothing, the server vanished from
# the account and clients reported "not authorized". The dashed name is public DNS that resolves
# to the embedded address, so it points at the LAN VIP AND matches the cert.
#
# WHY IT IS A WHOLE URL AND NOT BUILT FROM ${PLEX_L4_VIP}: postBuild substitution is plain
# envsubst — it cannot turn 192.168.2.24 into 192-168-2-24 — and `<serverhash>` is this server's
# certificate identity, derivable from nothing in this tree. Re-read it from the live cert
# (`openssl s_client -connect <vip>:32400 | openssl x509 -noout -subject`) if Plex ever reissues.
#
# The dashed IP here MUST track ${PLEX_L4_VIP} above. They are two halves of one address and
# nothing checks that they agree.
#
# dev -> PLACEHOLDER, non-functional. dev is decommissioned and has no claimed Plex server, so
# there is no real hash to put here; the base default only has to be a defined, correctly
# shaped string (StrictPostBuildSubstitutions makes an UNDEFINED var a hard failure).
# prod -> the real 192.168.2.24 + live server hash; see clusters/prod/kustomization.yaml.
PLEX_ADVERTISE_URL: "https://192-168-1-57.0000000000000000000000000000000f.plex.direct:32400"
# The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress
# peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable
# from `world` is an open resolver and an amplification reflector, so the allow is scoped to this
clusters/prod/kustomization.yaml
+10
View File
@@ -496,6 +496,16 @@ patches:
- op: replace
path: /data/PLEX_L4_VIP
value: "192.168.2.24"
# PLEX_ADVERTISE_URL — the prod VIP above in Plex's `plex.direct` form, which is the ONLY form
# that validates: the server's cert is `*.130072f20c334d6c8f63d0dfef87309b.plex.direct` with no
# IP SAN (read off the live listener 2026-10-04), so a bare `https://192.168.2.24:32400` fails
# TLS for every client — the 2026-10-04 outage. Measured with full verification, no `curl -k`:
# this URL answers /identity with 200 and TLS verify OK; the bare-IP form fails to connect.
# THE DASHED IP MUST MATCH /data/PLEX_L4_VIP ABOVE (192.168.2.24 -> 192-168-2-24); nothing
# enforces that. The hash is server identity, not a secret, and changes only if Plex reissues.
- op: replace
path: /data/PLEX_ADVERTISE_URL
value: "https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400"
# The prod flat LAN. This is the 192.168.2.0/24 segment the SOHO ROUTER at 192.168.2.1 owns —
# NOT pfSense's 192.168.1.0/24, which is a different network and not in the LAN-DNS path.
# It scopes the ingress allow for PiHole's DNS listener to LAN clients only.
decisions/2026-10-04-plex-advertise-plexdirect.md
+139
View File
@@ -0,0 +1,139 @@
---
date: 2026-10-04
branch: fix/plex-advertise-plexdirect
title: Plex vanished from the plex.tv account — every advertised connection was uncompletable; advertise the plex.direct name instead of the bare VIP
---
## What changed
Reported symptom: the server no longer appears on the operator's plex.tv account; `plex.home.keithjackson.dev`
shows "not authorized"; connecting straight to the VIP + port works.
**Root cause, measured: `customConnections` held two hand-typed URLs and NEITHER could ever complete,
so Plex had nothing publishable.** Because `customConnections` OVERRIDES Plex's own connection
publishing, the server ended up advertising only dead endpoints. Live `Preferences.xml`, 2026-10-04:
customConnections="https://plex.home.keithjackson.dev:32400/,https://192.168.2.24:32400/"
ManualPortMappingMode="1" ManualPortMappingPort=<ABSENT>
- `https://192.168.2.24:32400` — **TLS can never validate.** Plex serves 32400 with
`CN=*.130072f20c334d6c8f63d0dfef87309b.plex.direct` and **no IP SAN**, so a bare-IP https URL fails
for every client. Not theoretical: Plex's log carries a continuous storm of
`CERT: incomplete TLS handshake from <node> ... sslv3 alert certificate unknown`, every few seconds,
from **real LAN clients** arriving through the new `plex-l4-tcp` VIP.
- `https://plex.home.keithjackson.dev:32400` — **nothing listens.** The shared Gateway serves that host
on 443; measured `http=000` (connect failed) against `192.168.2.20:32400`.
Consequence chain, each link measured: no completable connection -> plex.tv could publish nothing
(`MyPlex: mapping state set to 'Mapped - Not Published'`, repeatedly) -> Plex fell back to **Plex Relay**
(`startRelay`, remote forward to 127.0.0.1:32401) -> **the server disappeared from the account**, and
clients reported "not authorized".
The fix advertises Plex's own `<dashed-ip>.<serverhash>.plex.direct` form, which is public DNS resolving
to the embedded address — so it points at the LAN VIP *and* matches the wildcard cert.
- **`catalog/plex.yaml`:** `ADVERTISE_IP: ${PLEX_ADVERTISE_URL}` added, replacing the long-standing
DEFERRAL note. The deferral had weighed only the L7 Gateway URL; the operator chose the native L4 path
on 2026-10-03, so the open question was never "Gateway or not" but "which URL has a valid cert".
- **`clusters/base/cluster-vars.yaml`:** new `PLEX_ADVERTISE_URL` knob (+ substitution-inventory comment
updated to name `plex: PLEX_ADVERTISE_URL`). The base default is an explicit **non-functional
placeholder** — dev is decommissioned and has no claimed server, and
`StrictPostBuildSubstitutions` makes an undefined var a hard failure, so the default only has to exist
and be correctly shaped.
- **`clusters/prod/kustomization.yaml`:** prod value
`https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400`.
- **`apps/wave-0/plex.yaml`:** re-rendered with the canonical command
(`python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation`). Two added
lines, nothing else.
**Why a whole-URL cluster-var and not something built from `${PLEX_L4_VIP}`:** postBuild substitution is
plain envsubst and cannot turn `192.168.2.24` into `192-168-2-24`, and `<serverhash>` is the server's
certificate identity, derivable from nothing in this tree. The dashed IP and `${PLEX_L4_VIP}` are two
halves of one address and **nothing checks that they agree** — noted in both files.
**SENSITIVE PATHS — this PR touches `clusters/**` (two files), so `ci/approval-policy.sh` will BLOCK it.**
It needs the operator's `/approve-override <reason>` posted as a PR **review** of type "Comment".
Deliberately not posted here: CLAUDE.md §1 forbids an agent clearing this on its own PR, and the agent
account holds `owner`, so the prohibition is a rule rather than a limit.
## What I verified
All measurements against live prod, 2026-10-04, read-only. Secrets handled per CLAUDE.md §6: preference
attributes were read by name-scoped grep, never by dumping `Preferences.xml` (it holds `PlexOnlineToken`),
and the token was reported only as `<PRESENT,len=22>`.
- **THE FIX ITSELF, with full TLS verification (no `curl -k`)** — the one measurement the whole change
rests on:
- `https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400/identity` -> **`http=200`, TLS verify OK**
- `https://192.168.2.24:32400/identity` -> **`http=000`, cert validation FAILED**
- that name resolves to `192.168.2.24` (`getent hosts`), confirming the dashed-IP mechanism works from
inside this network and is not blocked by DNS-rebinding protection on the resolver path tested.
- **The server is still claimed — this was never a lost claim.** `/identity` -> `claimed="1"` with the real
`machineIdentifier`; `PlexOnlineToken` present, `PlexOnlineUsername="jakman85"`.
- **The certificate, read off the live listener:** exactly one SAN,
`DNS:*.130072f20c334d6c8f63d0dfef87309b.plex.direct`. No IP SAN. This is why the bare-IP URL fails.
- **The publish payload Plex actually sent to plex.tv**, which is what proves the connection list was the
problem rather than a guess: `PUT https://servers.plex.tv/devices/<id>?Connection[][uri]=https://plex.home.keithjackson.dev:32400&Connection[][uri]=https://192.168.2.24:32400&Connection[][uri]=http://136.51.115.252:32400&Connection[][uri]=http://10.0.2.136:32400&httpsEnabled=1...`
— all four unreachable from a LAN client or from plex.tv.
- **`Mapped - Not Published`** at every reachability check across the whole log, and an active **Plex Relay**
session — the documented fallback when nothing is directly publishable.
- **The `certificate unknown` TLS failures come from real client traffic, not a probe.** The source
`10.0.1.78` resolves via `cilium-dbg bpf ipcache get` to **identity 1 = `reserved:host`**, i.e. LAN
clients SNAT'd through `plex-l4-tcp` (`externalTrafficPolicy: Cluster`) on prod-3.
- **The Gateway is NOT broken, and my first reading of it was wrong.** A 403 `Access denied` from
`server: envoy` on the Gateway path turned out to be an artifact of probing from inside the plex pod:
**all 16** Gateway hostnames return 403 from there, because Cilium's Gateway preserves the original
source identity and enforces the destination's ingress policy. Plex's own log shows a concurrent,
**working, signed-in** browser session through the Gateway returning 200s. The HTTPRoute is
`Accepted=True`/`ResolvedRefs=True` with no filters, and it is the only route claiming that hostname.
- **Cilium is not denying the Gateway->Plex hop:** `cilium-dbg bpf policy get 3796` shows
`Allow Ingress reserved:ingress 32400/TCP`, PROXY PORT `NONE`, with 141k packets counted. No L7/HTTP
rules and no CCNPs exist anywhere in the cluster.
- **In-cluster consumers were never affected:** Seerr's recently-added scans and watchlist syncs complete
cleanly with no auth errors; there are **zero** HTTP 401s in Plex's log (the five ` 401 ` hits are
transcoder segment numbers).
- **`ADVERTISE_IP`'s mechanism, in the running image rather than from docs:**
`/etc/cont-init.d/40-plex-first-run` line 127 does `setPref "customConnections" "${ADVERTISE_IP}"`,
outside the claim guard, and `/.firstRunComplete` (lines 9/139) is on the container overlay, not the
`/config` PVC — so it re-applies on every pod start. Editing the env rolls the Deployment, which applies it.
- **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with
`spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all
**pass**. The rc 3 is the two known environmental skips — `gitea-settings` (no local
`SETTINGS_API_TOKEN`) and `conformance` L2/L3 (`CLUSTER_AVAILABLE` unset). **Not green. Reported as rc 3.**
- **`render-golden` passing is the specific proof** that the catalog edit and the committed golden agree;
the rendered diff is exactly two lines and no other service's manifest moved.
## What I did NOT verify
- **THE OUTCOME. Nothing here has been observed to put the server back on the plex.tv account.** The
change is unreconciled: `ADVERTISE_IP` is absent from the live Deployment, the live
`customConnections` still holds the two broken URLs, and Plex has not been restarted. What is proven is
that the *new URL answers 200 with a valid certificate* and that *every currently advertised URL cannot
complete* — a necessary-and-plausibly-sufficient argument, not a demonstrated fix.
- **Whether plex.tv will publish and list the server once this lands.** plex.tv's decision is made
off-network and cannot be measured from here. A LAN-only connection may still leave the account showing
the server as remotely unreachable even when local clients work.
- **REMOTE (off-LAN) ACCESS IS NOT FIXED BY THIS, and may still be fully broken.** The advertised URL
resolves to an RFC1918 address, so it serves LAN clients only. The off-repo WAN tcp/32400 port-forward
(`runbooks/plex-remote-access.md`) is still **unverified** — `http://136.51.115.252:32400` was
unreachable from inside the LAN, but 2026-10-03 established there is no NAT hairpin here, so that is
**inconclusive**, not proof of absence. This repo has no off-LAN vantage point.
- **`ManualPortMappingMode="1"` with `ManualPortMappingPort` ABSENT is left as-is.** Manual public-port
mapping is enabled with no port set — a half-configured state I did not change and whose exact effect on
publishing I did not isolate. There is **no GitOps path** to it (2026-10-03: the image supports no
`PLEX_PREFERENCE*` of any kind), so it is an operator UI step.
- **Whether the operator intended the hand-typed URLs** to override the 2026-10-03 native-L4 design, or
typed them while debugging. I did not alter live preferences; the env var will overwrite
`customConnections` on the next pod start, which is a deliberate transfer of ownership from the web form
to git and will **silently discard** anything typed there since.
- **The `plex.direct` hash's stability.** Treated as server identity and read once from the live cert. I
did not establish what reissues it, so this URL could go stale with no alert; nothing in CI or the
conformance suite checks that the dashed IP still matches `${PLEX_L4_VIP}` or that the hash is current.
- **Resolver behaviour for the whole LAN.** `plex.direct` resolved correctly from a *pod* (cluster DNS
path). LAN clients resolve via PiHole, and dnsmasq's `stop-dns-rebind` would reject a public answer
carrying a private address. I did **not** test resolution from a LAN client, and this is the most likely
way the fix still fails.
- **The CI approval-policy outcome.** The local stage reported green, but that is an offline N/A, not a
verdict: this PR touches `clusters/**`, so CI will block pending the operator's override.
- **Why PMS restarted.** The process came up at 05:26 UTC while the pod is 2d2h old with 0 restarts, so
something restarted Plex inside the container. Unexplained and not investigated.