home-network-k3s-impl/runbooks/plex-remote-access.md

plex-remote-access — getting Plex reachable from outside the LAN, and proving it

Status: the in-cluster half is shipped; the last mile is TWO MANUAL STEPS and remote access does NOT work until both are done. Neither can live in this repo. Alert: none, and none is possible from in-cluster signals — see "Why no alert" below. This is an operator procedure, not an incident page. Files: apps/wave-0/plex-remote.yaml (the tcp/32400 LoadBalancer + its CiliumNetworkPolicy), clusters/base/cluster-vars.yaml + clusters/prod/kustomization.yaml (PLEX_L4_VIP), catalog/render.py (the EXTRA_UNITS entry that places the unit), catalog/plex.yaml (Plex's own env — note ADVERTISE_IP is deliberately NOT set; see "Why not the Gateway URL"). Related: runbooks/plex-quicksync.md and runbooks/plex-claim.md — the other two Plex procedures with a manual last mile.

The Plex web UI is LAN/VPN-only (2026-11-08)

When public DNS moved to the Cloudflare Tunnel (runbooks/cloudflare-tunnel.md, Phase 2), the operator kept Plex off the tunnel. catalog/plex.yaml is now exposure: internal, so the plex.home.keithjackson.dev route carries external-dns ... controller: none and its public record is deleted; the tunnel also 404s plex and plex-vip. On the LAN (and over WireGuard) the web UI at https://plex.home.keithjackson.dev works exactly as before through PiHole's split-horizon answer.

Remote streaming is unaffected — it never used that name. It is the path below: plex.tv hands clients the plex.direct URL (ADVERTISE_IP) for the L4 VIP 192.168.2.24, reached through the WAN tcp/32400 forward. That forward must stay open when the WAN 80/443 forwards are closed after the tunnel cut-over. Remote users use app.plex.tv, not our hostname.

What was actually broken (measured 2026-10-03)

Plex was serving correctly and was reachable on the LAN. Two independent things stopped remote clients:

1. Plex advertised no remote connection at all. In the live Preferences.xml: customConnections ABSENT, ManualPortMappingMode ABSENT, while PublishServerOnPlexOnlineKey="1". The server was trying to publish itself to plex.tv and had nothing reachable to publish. Plex's automatic path (UPnP/NAT-PMP) cannot work here: the process lives in a pod on the Cilium overlay (10.0.2.136), it cannot speak UPnP to the SOHO router, and a mapping to the pod IP would be meaningless if it could.
2. Nothing from the WAN reached the cluster. 136.51.115.252:32400 → connection refused. 136.51.115.252:443 → a self-signed router certificate (C=ZZ, CN=%D), not the Gateway's Let's Encrypt wildcard.

The Service was ClusterIP only, so port 32400 had no path off-cluster by any route.

Caveat on that second measurement, because it is easy to over-read: it was taken from inside the LAN, so strictly it proves there is no NAT hairpin (reflection). It does not prove outside-in failure. But a router certificate answering 443 means the router's own management UI is bound to the WAN port, and that would shadow a port-forward for genuine outside traffic too. Verify from a real outside vantage point (phone on cellular, or an external port checker) before concluding either way — this repo has no such vantage point and must not pretend otherwise.

Step 1 — forward WAN tcp/32400 on the router (operator, off-repo)

WAN tcp/32400  ->  192.168.2.24  (${PLEX_L4_VIP})  port 32400

192.168.2.24 is a pinned Cilium LB VIP, ARP-announced on the flat LAN by the existing CiliumL2AnnouncementPolicy default-l2. It is pinned on purpose: this address is typed into the router by hand, so it must never move on a reschedule.

While in the router, check what else is bound to the WAN interface. If remote management is listening on 443 (see the certificate measurement above), move it or disable it. (Historical: this used to shadow the Gateway's 80/443 forward for the Plex web UI over the internet. Since 2026-11-08 the web UI is LAN/VPN-only and public HTTP(S) goes through the Cloudflare Tunnel, so the 80/443 forwards are retired; only tcp/32400 matters here.)

Confirm the VIP is actually answering on the LAN first, so a failure later is unambiguous:

./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex get svc plex-l4-tcp -o wide'

EXTERNAL-IP must read 192.168.2.24. If it is <pending>, LB-IPAM could not assign — the address is outside the pool or already taken; see clusters/prod/kustomization.yaml for the pool accounting.

192.168.2.24 was never ARP-probed off-cluster. Only the in-cluster pool accounting was checked (IPsTotal 11 / IPsUsed 4 / IPsAvailable 7 on 2026-10-03). If the VIP behaves oddly, probe it from a LAN host — arping 192.168.2.24 — the way .23 was probed before it was claimed.

Step 2 — tick "manually specify public port" in Plex (operator, UI only)

There is no GitOps path to this and no environment variable for it. Measured in the running image on 2026-10-03: a grep of the entire init surface (/etc/cont-init.d, /plex-common.sh, /usr/local/bin) for ManualPortMapping and PLEX_PREFERENCE returned NONE. The image writes only six preferences, and this is not one of them. It is the same class of manual last mile as the HardwareAcceleratedCodecs toggle in runbooks/plex-quicksync.md.

In the Plex web UI (https://plex.home.keithjackson.dev, on the LAN):

Settings → Remote Access

• Ensure Enable Remote Access is on. (PublishServerOnPlexOnlineKey was already 1.)
• Tick Manually specify public port and set it to 32400.
• Apply, then let the panel settle. It should report "Fully accessible outside your network".

Without this tick Plex waits for a NAT mapping it can never obtain and publishes no remote connection, so the port-forward in step 1 sits there unused.

How to verify

Do these in order; each one isolates a different half.

1. The port is open from outside. From a phone on cellular, or any off-LAN host:

curl -sk -o /dev/null -w '%{http_code}\n' https://136.51.115.252:32400/identity

200 means the forward and the policy are both good. A hang or refusal means step 1.

2. Plex itself believes it is reachable. Settings → Remote Access must say "Fully accessible outside your network". If it says "not available outside your network" while test 1 passes, step 2 was not applied.

3. plex.tv is handing out a remote connection. The decisive test, because it is what clients actually consume: sign in to app.plex.tv from an off-LAN network and confirm the server appears and plays. A client finds this server through plex.tv's connection list, never through DNS.

4. The policy is admitting the traffic. If test 1 fails after the forward is confirmed, the CiliumNetworkPolicy is the next suspect:

./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex get cnp plex-allow-remote-32400'

See runbooks/denied-flows.md for reading actual drops.

Why no alert

Every in-cluster signal is green whether or not remote access works. The Service has endpoints, the pod is Running, the policy is applied, and plex-down watches kube_deployment_status_replicas_available — none of which can see a router with a missing port-forward or an unticked checkbox. An alert would need an off-LAN prober, which this cluster does not have. Test 3 above is the only honest check, and it is manual.

Why not the Gateway URL (ADVERTISE_IP)

The alternative was to leave Plex on ClusterIP and set ADVERTISE_IP: https://plex.home.keithjackson.dev:443, which writes Plex's customConnections preference and would have put the server on plex.tv's list via the existing L7 Gateway — no new WAN hole. That mechanism was verified to work on an already-claimed server (the ADVERTISE_IP block at line 126 of the image's 40-plex-first-run sits outside the claim guard, and /.firstRunComplete lives on the overlay FS, not the /config PVC, so it re-runs every pod start), and the Gateway path itself was measured serving Plex correctly: HTTP 200, claimed="1", with 401 on every protected endpoint.

The operator chose the native path anyway (2026-10-03), and the reasons are real: Plex terminates its own TLS with its *.plex.direct certificate, and video streams bypass the Envoy L7 hop instead of being proxied through it. The cost is a new unfiltered WAN port — acceptable because Plex authenticates every caller itself. ADVERTISE_IP therefore stays unset in catalog/plex.yaml, and the deferral note there is still accurate.

Known wart: remote clients are classified as LAN

apps/wave-0/plex-remote.yaml uses externalTrafficPolicy: Cluster, which SNATs the client to the ingress node's address (192.168.2.1x). That falls inside the 192.168.0.0/16 entry of ALLOWED_NETWORKS, so Plex treats every remote stream as local. Measured on the Gateway path, which SNATs the same way: Plex logged [10.0.2.214:36040 (Allowed Network (WAN))].

This is a metering effect, not a security one — authorization is unchanged, as the 401s above show — but Plex's remote-quality and bandwidth limits will not bind on these streams.

Local would preserve the client IP and fix it. Do not switch to it. Cilium's L2 announcer elects one lease holder per Service and does not follow the backend pod, so the VIP can be announced by a node with no Plex pod, black-holing the port while everything reports healthy. That exact change cost this LAN an 8-day Minecraft outage in September 2026 — the full write-up is at the top of platform/crafty/service-l4.yaml. Plex is a single-replica Recreate Deployment, so the node mismatch is the normal case here, not a rare one.