home-network-k3s-impl/runbooks/plex-remote-access.md

# plex-remote-access — getting Plex reachable from outside the LAN, and proving it

**Status: the in-cluster half is shipped; the last mile is TWO MANUAL STEPS and remote access does
NOT work until both are done.** Neither can live in this repo.
**Alert:** none, and none is possible from in-cluster signals — see "Why no alert" below. This is an
**operator procedure**, not an incident page.
**Files:** `apps/wave-0/plex-remote.yaml` (the tcp/32400 LoadBalancer + its CiliumNetworkPolicy),
`clusters/base/cluster-vars.yaml` + `clusters/prod/kustomization.yaml` (`PLEX_L4_VIP`),
`catalog/render.py` (the `EXTRA_UNITS` entry that places the unit), `catalog/plex.yaml` (Plex's own
env — note `ADVERTISE_IP` is deliberately NOT set; see "Why not the Gateway URL").
**Related:** `runbooks/plex-quicksync.md` and `runbooks/plex-claim.md` — the other two Plex
procedures with a manual last mile.

## The Plex web UI is LAN/VPN-only (2026-11-08)

When public DNS moved to the Cloudflare Tunnel (`runbooks/cloudflare-tunnel.md`, Phase 2), the
operator kept **Plex off the tunnel**. `catalog/plex.yaml` is now `exposure: internal`, so the
`plex.home.keithjackson.dev` route carries `external-dns ... controller: none` and its public record
is deleted; the tunnel also 404s `plex` and `plex-vip`. On the LAN (and over WireGuard) the web UI
at `https://plex.home.keithjackson.dev` works exactly as before through PiHole's split-horizon answer.

**Remote streaming is unaffected** — it never used that name. It is the path below: plex.tv hands
clients the `plex.direct` URL (`ADVERTISE_IP`) for the L4 VIP `192.168.2.24`, reached through the
WAN **tcp/32400** forward. That forward must stay open when the WAN 80/443 forwards are closed after
the tunnel cut-over. Remote users use app.plex.tv, not our hostname.

## What was actually broken (measured 2026-10-03)

Plex was serving correctly and was reachable on the LAN. Two independent things stopped remote
clients:

1. **Plex advertised no remote connection at all.** In the live `Preferences.xml`:
   `customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while
   `PublishServerOnPlexOnlineKey="1"`. The server was trying to publish itself to plex.tv and had
   nothing reachable to publish. Plex's automatic path (UPnP/NAT-PMP) cannot work here: the process
   lives in a pod on the Cilium overlay (`10.0.2.136`), it cannot speak UPnP to the SOHO router, and
   a mapping to the pod IP would be meaningless if it could.
2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` → connection refused.
   `136.51.115.252:443` → a **self-signed router certificate** (`C=ZZ, CN=%D`), not the Gateway's
   Let's Encrypt wildcard.

The Service was `ClusterIP` only, so port 32400 had no path off-cluster by any route.

**Caveat on that second measurement, because it is easy to over-read:** it was taken from *inside*
the LAN, so strictly it proves there is no NAT hairpin (reflection). It does not prove outside-in
failure. But a router certificate answering 443 means the router's own management UI is bound to the
WAN port, and that *would* shadow a port-forward for genuine outside traffic too. **Verify from a
real outside vantage point** (phone on cellular, or an external port checker) before concluding
either way — this repo has no such vantage point and must not pretend otherwise.

## Step 1 — forward WAN tcp/32400 on the router (operator, off-repo)

    WAN tcp/32400  ->  192.168.2.24  (${PLEX_L4_VIP})  port 32400

`192.168.2.24` is a pinned Cilium LB VIP, ARP-announced on the flat LAN by the existing
`CiliumL2AnnouncementPolicy default-l2`. It is **pinned on purpose**: this address is typed into the
router by hand, so it must never move on a reschedule.

While in the router, **check what else is bound to the WAN interface.** If remote management is
listening on 443 (see the certificate measurement above), move it or disable it. (Historical: this
used to shadow the Gateway's 80/443 forward for the Plex *web UI* over the internet. Since
2026-11-08 the web UI is LAN/VPN-only and public HTTP(S) goes through the Cloudflare Tunnel, so the
80/443 forwards are retired; only tcp/32400 matters here.)

Confirm the VIP is actually answering on the LAN first, so a failure later is unambiguous:

    ./infra/devcontainer/dev.sh run bash -c \
      'kubectl --context k3s-prod -n plex get svc plex-l4-tcp -o wide'

`EXTERNAL-IP` must read `192.168.2.24`. If it is `<pending>`, LB-IPAM could not assign — the address
is outside the pool or already taken; see `clusters/prod/kustomization.yaml` for the pool accounting.

**`192.168.2.24` was never ARP-probed off-cluster.** Only the in-cluster pool accounting was checked
(IPsTotal 11 / IPsUsed 4 / IPsAvailable 7 on 2026-10-03). If the VIP behaves oddly, probe it from a
LAN host — `arping 192.168.2.24` — the way `.23` was probed before it was claimed.

## Step 2 — tick "manually specify public port" in Plex (operator, UI only)

**There is no GitOps path to this and no environment variable for it.** Measured in the running
image on 2026-10-03: a grep of the entire init surface (`/etc/cont-init.d`, `/plex-common.sh`,
`/usr/local/bin`) for `ManualPortMapping` and `PLEX_PREFERENCE` returned **NONE**. The image writes
only six preferences, and this is not one of them. It is the same class of manual last mile as the
`HardwareAcceleratedCodecs` toggle in `runbooks/plex-quicksync.md`.

In the Plex web UI (`https://plex.home.keithjackson.dev`, on the LAN):

**Settings → Remote Access**
- Ensure **Enable Remote Access** is on. (`PublishServerOnPlexOnlineKey` was already `1`.)
- Tick **Manually specify public port** and set it to **32400**.
- **Apply**, then let the panel settle. It should report *"Fully accessible outside your network"*.

Without this tick Plex waits for a NAT mapping it can never obtain and publishes no remote
connection, so the port-forward in step 1 sits there unused.

## How to verify

Do these in order; each one isolates a different half.

**1. The port is open from outside.** From a phone on cellular, or any off-LAN host:

    curl -sk -o /dev/null -w '%{http_code}\n' https://136.51.115.252:32400/identity

`200` means the forward and the policy are both good. A hang or refusal means step 1.

**2. Plex itself believes it is reachable.** Settings → Remote Access must say *"Fully accessible
outside your network"*. If it says *"not available outside your network"* while test 1 passes, step 2
was not applied.

**3. plex.tv is handing out a remote connection.** The decisive test, because it is what clients
actually consume: sign in to [app.plex.tv](https://app.plex.tv) from an off-LAN network and confirm
the server appears and plays. A client finds this server through plex.tv's connection list, never
through DNS.

**4. The policy is admitting the traffic.** If test 1 fails *after* the forward is confirmed, the
CiliumNetworkPolicy is the next suspect:

    ./infra/devcontainer/dev.sh run bash -c \
      'kubectl --context k3s-prod -n plex get cnp plex-allow-remote-32400'

See `runbooks/denied-flows.md` for reading actual drops.

## Why no alert

Every in-cluster signal is green whether or not remote access works. The Service has endpoints, the
pod is `Running`, the policy is applied, and `plex-down` watches
`kube_deployment_status_replicas_available` — none of which can see a router with a missing
port-forward or an unticked checkbox. An alert would need an off-LAN prober, which this cluster does
not have. **Test 3 above is the only honest check, and it is manual.**

## Why not the Gateway URL (`ADVERTISE_IP`)

The alternative was to leave Plex on `ClusterIP` and set
`ADVERTISE_IP: https://plex.home.keithjackson.dev:443`, which writes Plex's `customConnections`
preference and would have put the server on plex.tv's list via the existing L7 Gateway — no new WAN
hole. That mechanism was verified to work on an already-claimed server (the `ADVERTISE_IP` block at
line 126 of the image's `40-plex-first-run` sits **outside** the claim guard, and `/.firstRunComplete`
lives on the overlay FS, not the `/config` PVC, so it re-runs every pod start), and the Gateway path
itself was measured serving Plex correctly: `HTTP 200`, `claimed="1"`, with `401` on every protected
endpoint.

**The operator chose the native path anyway (2026-10-03), and the reasons are real:** Plex terminates
its own TLS with its `*.plex.direct` certificate, and video streams bypass the Envoy L7 hop instead
of being proxied through it. The cost is a new unfiltered WAN port — acceptable because Plex
authenticates every caller itself. `ADVERTISE_IP` therefore stays unset in `catalog/plex.yaml`, and
the deferral note there is still accurate.

## Known wart: remote clients are classified as LAN

`apps/wave-0/plex-remote.yaml` uses `externalTrafficPolicy: Cluster`, which SNATs the client to the
ingress node's address (`192.168.2.1x`). That falls inside the `192.168.0.0/16` entry of
`ALLOWED_NETWORKS`, so **Plex treats every remote stream as local.** Measured on the Gateway path,
which SNATs the same way: Plex logged `[10.0.2.214:36040 (Allowed Network (WAN))]`.

This is a **metering** effect, not a security one — authorization is unchanged, as the 401s above
show — but Plex's remote-quality and bandwidth limits will not bind on these streams.

`Local` would preserve the client IP and fix it. **Do not switch to it.** Cilium's L2 announcer
elects one lease holder per Service and does *not* follow the backend pod, so the VIP can be
announced by a node with no Plex pod, black-holing the port while everything reports healthy. That
exact change cost this LAN an 8-day Minecraft outage in September 2026 — the full write-up is at the
top of `platform/crafty/service-l4.yaml`. Plex is a single-replica `Recreate` Deployment, so the
node mismatch is the normal case here, not a rare one.