jakman85andClaude Opus 5
fbaeb3e038
Remote Plex clients had no path in, for two measured reasons: Plex advertised no remote connection at all (customConnections and ManualPortMappingMode both ABSENT while PublishServerOnPlexOnlineKey=1), and its Service was ClusterIP, so 32400 had no off-cluster path. The L7 Gateway was NOT the problem -- it serves Plex correctly (200, claimed=1, 401 on protected endpoints) -- but Plex clients discover servers through plex.tv, not DNS, so an HTTPRoute cannot put this server on a client's connection list. Operator chose the native path over ADVERTISE_IP: Plex terminates its own TLS with *.plex.direct and streams bypass the Envoy L7 hop. Adds a hand-written non-catalog unit (the SD schema cannot express a LoadBalancer Service or a pinned lbipam VIP), registered in render.py's EXTRA_UNITS so --emit-waves places it in plex's own wave. externalTrafficPolicy: Cluster, inheriting crafty's 8-day-outage lesson rather than re-deriving it. Known consequence, documented: SNAT puts remote clients inside ALLOWED_NETWORKS, so Plex meters them as LAN. Not an auth change. Remote access does NOT work until the operator does two manual steps that cannot live in this repo -- the WAN tcp/32400 forward and Plex's "manually specify public port" toggle (no env var exists for it; verified NONE in the image's init surface). runbooks/plex-remote-access.md is the procedure. Touches clusters/** -- a sensitive path -- so the approval-policy stage will block pending an operator /approve-override. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
8.7 KiB
plex-remote-access — getting Plex reachable from outside the LAN, and proving it
Status: the in-cluster half is shipped; the last mile is TWO MANUAL STEPS and remote access does
NOT work until both are done. Neither can live in this repo.
Alert: none, and none is possible from in-cluster signals — see "Why no alert" below. This is an
operator procedure, not an incident page.
Files: apps/wave-0/plex-remote.yaml (the tcp/32400 LoadBalancer + its CiliumNetworkPolicy),
clusters/base/cluster-vars.yaml + clusters/prod/kustomization.yaml (PLEX_L4_VIP),
catalog/render.py (the EXTRA_UNITS entry that places the unit), catalog/plex.yaml (Plex's own
env — note ADVERTISE_IP is deliberately NOT set; see "Why not the Gateway URL").
Related: runbooks/plex-quicksync.md and runbooks/plex-claim.md — the other two Plex
procedures with a manual last mile.
What was actually broken (measured 2026-10-03)
Plex was serving correctly and was reachable on the LAN. Two independent things stopped remote clients:
- Plex advertised no remote connection at all. In the live
Preferences.xml:customConnectionsABSENT,ManualPortMappingModeABSENT, whilePublishServerOnPlexOnlineKey="1". The server was trying to publish itself to plex.tv and had nothing reachable to publish. Plex's automatic path (UPnP/NAT-PMP) cannot work here: the process lives in a pod on the Cilium overlay (10.0.2.136), it cannot speak UPnP to the SOHO router, and a mapping to the pod IP would be meaningless if it could. - Nothing from the WAN reached the cluster.
136.51.115.252:32400→ connection refused.136.51.115.252:443→ a self-signed router certificate (C=ZZ, CN=%D), not the Gateway's Let's Encrypt wildcard.
The Service was ClusterIP only, so port 32400 had no path off-cluster by any route.
Caveat on that second measurement, because it is easy to over-read: it was taken from inside the LAN, so strictly it proves there is no NAT hairpin (reflection). It does not prove outside-in failure. But a router certificate answering 443 means the router's own management UI is bound to the WAN port, and that would shadow a port-forward for genuine outside traffic too. Verify from a real outside vantage point (phone on cellular, or an external port checker) before concluding either way — this repo has no such vantage point and must not pretend otherwise.
Step 1 — forward WAN tcp/32400 on the router (operator, off-repo)
WAN tcp/32400 -> 192.168.2.24 (${PLEX_L4_VIP}) port 32400
192.168.2.24 is a pinned Cilium LB VIP, ARP-announced on the flat LAN by the existing
CiliumL2AnnouncementPolicy default-l2. It is pinned on purpose: this address is typed into the
router by hand, so it must never move on a reschedule.
While in the router, check what else is bound to the WAN interface. If remote management is listening on 443 (see the certificate measurement above), move it or disable it — otherwise the Gateway's 80/443 forward is shadowed and the Plex web UI over the internet stays broken even though remote streaming works. These are two separate paths and they fail independently.
Confirm the VIP is actually answering on the LAN first, so a failure later is unambiguous:
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex get svc plex-l4-tcp -o wide'
EXTERNAL-IP must read 192.168.2.24. If it is <pending>, LB-IPAM could not assign — the address
is outside the pool or already taken; see clusters/prod/kustomization.yaml for the pool accounting.
192.168.2.24 was never ARP-probed off-cluster. Only the in-cluster pool accounting was checked
(IPsTotal 11 / IPsUsed 4 / IPsAvailable 7 on 2026-10-03). If the VIP behaves oddly, probe it from a
LAN host — arping 192.168.2.24 — the way .23 was probed before it was claimed.
Step 2 — tick "manually specify public port" in Plex (operator, UI only)
There is no GitOps path to this and no environment variable for it. Measured in the running
image on 2026-10-03: a grep of the entire init surface (/etc/cont-init.d, /plex-common.sh,
/usr/local/bin) for ManualPortMapping and PLEX_PREFERENCE returned NONE. The image writes
only six preferences, and this is not one of them. It is the same class of manual last mile as the
HardwareAcceleratedCodecs toggle in runbooks/plex-quicksync.md.
In the Plex web UI (https://plex.home.keithjackson.dev, on the LAN):
Settings → Remote Access
- Ensure Enable Remote Access is on. (
PublishServerOnPlexOnlineKeywas already1.) - Tick Manually specify public port and set it to 32400.
- Apply, then let the panel settle. It should report "Fully accessible outside your network".
Without this tick Plex waits for a NAT mapping it can never obtain and publishes no remote connection, so the port-forward in step 1 sits there unused.
How to verify
Do these in order; each one isolates a different half.
1. The port is open from outside. From a phone on cellular, or any off-LAN host:
curl -sk -o /dev/null -w '%{http_code}\n' https://136.51.115.252:32400/identity
200 means the forward and the policy are both good. A hang or refusal means step 1.
2. Plex itself believes it is reachable. Settings → Remote Access must say "Fully accessible outside your network". If it says "not available outside your network" while test 1 passes, step 2 was not applied.
3. plex.tv is handing out a remote connection. The decisive test, because it is what clients actually consume: sign in to app.plex.tv from an off-LAN network and confirm the server appears and plays. A client finds this server through plex.tv's connection list, never through DNS.
4. The policy is admitting the traffic. If test 1 fails after the forward is confirmed, the CiliumNetworkPolicy is the next suspect:
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex get cnp plex-allow-remote-32400'
See runbooks/denied-flows.md for reading actual drops.
Why no alert
Every in-cluster signal is green whether or not remote access works. The Service has endpoints, the
pod is Running, the policy is applied, and plex-down watches
kube_deployment_status_replicas_available — none of which can see a router with a missing
port-forward or an unticked checkbox. An alert would need an off-LAN prober, which this cluster does
not have. Test 3 above is the only honest check, and it is manual.
Why not the Gateway URL (ADVERTISE_IP)
The alternative was to leave Plex on ClusterIP and set
ADVERTISE_IP: https://plex.home.keithjackson.dev:443, which writes Plex's customConnections
preference and would have put the server on plex.tv's list via the existing L7 Gateway — no new WAN
hole. That mechanism was verified to work on an already-claimed server (the ADVERTISE_IP block at
line 126 of the image's 40-plex-first-run sits outside the claim guard, and /.firstRunComplete
lives on the overlay FS, not the /config PVC, so it re-runs every pod start), and the Gateway path
itself was measured serving Plex correctly: HTTP 200, claimed="1", with 401 on every protected
endpoint.
The operator chose the native path anyway (2026-10-03), and the reasons are real: Plex terminates
its own TLS with its *.plex.direct certificate, and video streams bypass the Envoy L7 hop instead
of being proxied through it. The cost is a new unfiltered WAN port — acceptable because Plex
authenticates every caller itself. ADVERTISE_IP therefore stays unset in catalog/plex.yaml, and
the deferral note there is still accurate.
Known wart: remote clients are classified as LAN
apps/wave-0/plex-remote.yaml uses externalTrafficPolicy: Cluster, which SNATs the client to the
ingress node's address (192.168.2.1x). That falls inside the 192.168.0.0/16 entry of
ALLOWED_NETWORKS, so Plex treats every remote stream as local. Measured on the Gateway path,
which SNATs the same way: Plex logged [10.0.2.214:36040 (Allowed Network (WAN))].
This is a metering effect, not a security one — authorization is unchanged, as the 401s above show — but Plex's remote-quality and bandwidth limits will not bind on these streams.
Local would preserve the client IP and fix it. Do not switch to it. Cilium's L2 announcer
elects one lease holder per Service and does not follow the backend pod, so the VIP can be
announced by a node with no Plex pod, black-holing the port while everything reports healthy. That
exact change cost this LAN an 8-day Minecraft outage in September 2026 — the full write-up is at the
top of platform/crafty/service-l4.yaml. Plex is a single-replica Recreate Deployment, so the
node mismatch is the normal case here, not a rare one.