jakman85andClaude Opus 5
fbaeb3e038
Remote Plex clients had no path in, for two measured reasons: Plex advertised no remote connection at all (customConnections and ManualPortMappingMode both ABSENT while PublishServerOnPlexOnlineKey=1), and its Service was ClusterIP, so 32400 had no off-cluster path. The L7 Gateway was NOT the problem -- it serves Plex correctly (200, claimed=1, 401 on protected endpoints) -- but Plex clients discover servers through plex.tv, not DNS, so an HTTPRoute cannot put this server on a client's connection list. Operator chose the native path over ADVERTISE_IP: Plex terminates its own TLS with *.plex.direct and streams bypass the Envoy L7 hop. Adds a hand-written non-catalog unit (the SD schema cannot express a LoadBalancer Service or a pinned lbipam VIP), registered in render.py's EXTRA_UNITS so --emit-waves places it in plex's own wave. externalTrafficPolicy: Cluster, inheriting crafty's 8-day-outage lesson rather than re-deriving it. Known consequence, documented: SNAT puts remote clients inside ALLOWED_NETWORKS, so Plex meters them as LAN. Not an auth change. Remote access does NOT work until the operator does two manual steps that cannot live in this repo -- the WAN tcp/32400 forward and Plex's "manually specify public port" toggle (no env var exists for it; verified NONE in the image's init surface). runbooks/plex-remote-access.md is the procedure. Touches clusters/** -- a sensitive path -- so the approval-policy stage will block pending an operator /approve-override. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
149 lines
8.7 KiB
Markdown
149 lines
8.7 KiB
Markdown
# plex-remote-access — getting Plex reachable from outside the LAN, and proving it
|
|
|
|
**Status: the in-cluster half is shipped; the last mile is TWO MANUAL STEPS and remote access does
|
|
NOT work until both are done.** Neither can live in this repo.
|
|
**Alert:** none, and none is possible from in-cluster signals — see "Why no alert" below. This is an
|
|
**operator procedure**, not an incident page.
|
|
**Files:** `apps/wave-0/plex-remote.yaml` (the tcp/32400 LoadBalancer + its CiliumNetworkPolicy),
|
|
`clusters/base/cluster-vars.yaml` + `clusters/prod/kustomization.yaml` (`PLEX_L4_VIP`),
|
|
`catalog/render.py` (the `EXTRA_UNITS` entry that places the unit), `catalog/plex.yaml` (Plex's own
|
|
env — note `ADVERTISE_IP` is deliberately NOT set; see "Why not the Gateway URL").
|
|
**Related:** `runbooks/plex-quicksync.md` and `runbooks/plex-claim.md` — the other two Plex
|
|
procedures with a manual last mile.
|
|
|
|
## What was actually broken (measured 2026-10-03)
|
|
|
|
Plex was serving correctly and was reachable on the LAN. Two independent things stopped remote
|
|
clients:
|
|
|
|
1. **Plex advertised no remote connection at all.** In the live `Preferences.xml`:
|
|
`customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while
|
|
`PublishServerOnPlexOnlineKey="1"`. The server was trying to publish itself to plex.tv and had
|
|
nothing reachable to publish. Plex's automatic path (UPnP/NAT-PMP) cannot work here: the process
|
|
lives in a pod on the Cilium overlay (`10.0.2.136`), it cannot speak UPnP to the SOHO router, and
|
|
a mapping to the pod IP would be meaningless if it could.
|
|
2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` → connection refused.
|
|
`136.51.115.252:443` → a **self-signed router certificate** (`C=ZZ, CN=%D`), not the Gateway's
|
|
Let's Encrypt wildcard.
|
|
|
|
The Service was `ClusterIP` only, so port 32400 had no path off-cluster by any route.
|
|
|
|
**Caveat on that second measurement, because it is easy to over-read:** it was taken from *inside*
|
|
the LAN, so strictly it proves there is no NAT hairpin (reflection). It does not prove outside-in
|
|
failure. But a router certificate answering 443 means the router's own management UI is bound to the
|
|
WAN port, and that *would* shadow a port-forward for genuine outside traffic too. **Verify from a
|
|
real outside vantage point** (phone on cellular, or an external port checker) before concluding
|
|
either way — this repo has no such vantage point and must not pretend otherwise.
|
|
|
|
## Step 1 — forward WAN tcp/32400 on the router (operator, off-repo)
|
|
|
|
WAN tcp/32400 -> 192.168.2.24 (${PLEX_L4_VIP}) port 32400
|
|
|
|
`192.168.2.24` is a pinned Cilium LB VIP, ARP-announced on the flat LAN by the existing
|
|
`CiliumL2AnnouncementPolicy default-l2`. It is **pinned on purpose**: this address is typed into the
|
|
router by hand, so it must never move on a reschedule.
|
|
|
|
While in the router, **check what else is bound to the WAN interface.** If remote management is
|
|
listening on 443 (see the certificate measurement above), move it or disable it — otherwise the
|
|
Gateway's 80/443 forward is shadowed and the Plex *web UI* over the internet stays broken even
|
|
though remote *streaming* works. These are two separate paths and they fail independently.
|
|
|
|
Confirm the VIP is actually answering on the LAN first, so a failure later is unambiguous:
|
|
|
|
./infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n plex get svc plex-l4-tcp -o wide'
|
|
|
|
`EXTERNAL-IP` must read `192.168.2.24`. If it is `<pending>`, LB-IPAM could not assign — the address
|
|
is outside the pool or already taken; see `clusters/prod/kustomization.yaml` for the pool accounting.
|
|
|
|
**`192.168.2.24` was never ARP-probed off-cluster.** Only the in-cluster pool accounting was checked
|
|
(IPsTotal 11 / IPsUsed 4 / IPsAvailable 7 on 2026-10-03). If the VIP behaves oddly, probe it from a
|
|
LAN host — `arping 192.168.2.24` — the way `.23` was probed before it was claimed.
|
|
|
|
## Step 2 — tick "manually specify public port" in Plex (operator, UI only)
|
|
|
|
**There is no GitOps path to this and no environment variable for it.** Measured in the running
|
|
image on 2026-10-03: a grep of the entire init surface (`/etc/cont-init.d`, `/plex-common.sh`,
|
|
`/usr/local/bin`) for `ManualPortMapping` and `PLEX_PREFERENCE` returned **NONE**. The image writes
|
|
only six preferences, and this is not one of them. It is the same class of manual last mile as the
|
|
`HardwareAcceleratedCodecs` toggle in `runbooks/plex-quicksync.md`.
|
|
|
|
In the Plex web UI (`https://plex.home.keithjackson.dev`, on the LAN):
|
|
|
|
**Settings → Remote Access**
|
|
- Ensure **Enable Remote Access** is on. (`PublishServerOnPlexOnlineKey` was already `1`.)
|
|
- Tick **Manually specify public port** and set it to **32400**.
|
|
- **Apply**, then let the panel settle. It should report *"Fully accessible outside your network"*.
|
|
|
|
Without this tick Plex waits for a NAT mapping it can never obtain and publishes no remote
|
|
connection, so the port-forward in step 1 sits there unused.
|
|
|
|
## How to verify
|
|
|
|
Do these in order; each one isolates a different half.
|
|
|
|
**1. The port is open from outside.** From a phone on cellular, or any off-LAN host:
|
|
|
|
curl -sk -o /dev/null -w '%{http_code}\n' https://136.51.115.252:32400/identity
|
|
|
|
`200` means the forward and the policy are both good. A hang or refusal means step 1.
|
|
|
|
**2. Plex itself believes it is reachable.** Settings → Remote Access must say *"Fully accessible
|
|
outside your network"*. If it says *"not available outside your network"* while test 1 passes, step 2
|
|
was not applied.
|
|
|
|
**3. plex.tv is handing out a remote connection.** The decisive test, because it is what clients
|
|
actually consume: sign in to [app.plex.tv](https://app.plex.tv) from an off-LAN network and confirm
|
|
the server appears and plays. A client finds this server through plex.tv's connection list, never
|
|
through DNS.
|
|
|
|
**4. The policy is admitting the traffic.** If test 1 fails *after* the forward is confirmed, the
|
|
CiliumNetworkPolicy is the next suspect:
|
|
|
|
./infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n plex get cnp plex-allow-remote-32400'
|
|
|
|
See `runbooks/denied-flows.md` for reading actual drops.
|
|
|
|
## Why no alert
|
|
|
|
Every in-cluster signal is green whether or not remote access works. The Service has endpoints, the
|
|
pod is `Running`, the policy is applied, and `plex-down` watches
|
|
`kube_deployment_status_replicas_available` — none of which can see a router with a missing
|
|
port-forward or an unticked checkbox. An alert would need an off-LAN prober, which this cluster does
|
|
not have. **Test 3 above is the only honest check, and it is manual.**
|
|
|
|
## Why not the Gateway URL (`ADVERTISE_IP`)
|
|
|
|
The alternative was to leave Plex on `ClusterIP` and set
|
|
`ADVERTISE_IP: https://plex.home.keithjackson.dev:443`, which writes Plex's `customConnections`
|
|
preference and would have put the server on plex.tv's list via the existing L7 Gateway — no new WAN
|
|
hole. That mechanism was verified to work on an already-claimed server (the `ADVERTISE_IP` block at
|
|
line 126 of the image's `40-plex-first-run` sits **outside** the claim guard, and `/.firstRunComplete`
|
|
lives on the overlay FS, not the `/config` PVC, so it re-runs every pod start), and the Gateway path
|
|
itself was measured serving Plex correctly: `HTTP 200`, `claimed="1"`, with `401` on every protected
|
|
endpoint.
|
|
|
|
**The operator chose the native path anyway (2026-10-03), and the reasons are real:** Plex terminates
|
|
its own TLS with its `*.plex.direct` certificate, and video streams bypass the Envoy L7 hop instead
|
|
of being proxied through it. The cost is a new unfiltered WAN port — acceptable because Plex
|
|
authenticates every caller itself. `ADVERTISE_IP` therefore stays unset in `catalog/plex.yaml`, and
|
|
the deferral note there is still accurate.
|
|
|
|
## Known wart: remote clients are classified as LAN
|
|
|
|
`apps/wave-0/plex-remote.yaml` uses `externalTrafficPolicy: Cluster`, which SNATs the client to the
|
|
ingress node's address (`192.168.2.1x`). That falls inside the `192.168.0.0/16` entry of
|
|
`ALLOWED_NETWORKS`, so **Plex treats every remote stream as local.** Measured on the Gateway path,
|
|
which SNATs the same way: Plex logged `[10.0.2.214:36040 (Allowed Network (WAN))]`.
|
|
|
|
This is a **metering** effect, not a security one — authorization is unchanged, as the 401s above
|
|
show — but Plex's remote-quality and bandwidth limits will not bind on these streams.
|
|
|
|
`Local` would preserve the client IP and fix it. **Do not switch to it.** Cilium's L2 announcer
|
|
elects one lease holder per Service and does *not* follow the backend pod, so the VIP can be
|
|
announced by a node with no Plex pod, black-holing the port while everything reports healthy. That
|
|
exact change cost this LAN an 8-day Minecraft outage in September 2026 — the full write-up is at the
|
|
top of `platform/crafty/service-l4.yaml`. Plex is a single-replica `Recreate` Deployment, so the
|
|
node mismatch is the normal case here, not a rare one.
|