From fbaeb3e0386e624c5da6e9c513176dd40501c72f Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 3 Oct 2026 23:30:46 -0500 Subject: [PATCH] feat(plex): expose tcp/32400 for remote access via a pinned LoadBalancer VIP Remote Plex clients had no path in, for two measured reasons: Plex advertised no remote connection at all (customConnections and ManualPortMappingMode both ABSENT while PublishServerOnPlexOnlineKey=1), and its Service was ClusterIP, so 32400 had no off-cluster path. The L7 Gateway was NOT the problem -- it serves Plex correctly (200, claimed=1, 401 on protected endpoints) -- but Plex clients discover servers through plex.tv, not DNS, so an HTTPRoute cannot put this server on a client's connection list. Operator chose the native path over ADVERTISE_IP: Plex terminates its own TLS with *.plex.direct and streams bypass the Envoy L7 hop. Adds a hand-written non-catalog unit (the SD schema cannot express a LoadBalancer Service or a pinned lbipam VIP), registered in render.py's EXTRA_UNITS so --emit-waves places it in plex's own wave. externalTrafficPolicy: Cluster, inheriting crafty's 8-day-outage lesson rather than re-deriving it. Known consequence, documented: SNAT puts remote clients inside ALLOWED_NETWORKS, so Plex meters them as LAN. Not an auth change. Remote access does NOT work until the operator does two manual steps that cannot live in this repo -- the WAN tcp/32400 forward and Plex's "manually specify public port" toggle (no env var exists for it; verified NONE in the image's init surface). runbooks/plex-remote-access.md is the procedure. Touches clusters/** -- a sensitive path -- so the approval-policy stage will block pending an operator /approve-override. Co-Authored-By: Claude Opus 5 (1M context) --- apps/wave-0/kustomization.yaml | 1 + apps/wave-0/plex-remote.yaml | 126 +++++++++++++++ catalog/render.py | 18 +++ clusters/base/cluster-vars.yaml | 16 +- clusters/prod/kustomization.yaml | 20 +++ decisions/2026-10-03-plex-remote-access-l4.md | 129 +++++++++++++++ runbooks/plex-remote-access.md | 148 ++++++++++++++++++ 7 files changed, 457 insertions(+), 1 deletion(-) create mode 100644 apps/wave-0/plex-remote.yaml create mode 100644 decisions/2026-10-03-plex-remote-access-l4.md create mode 100644 runbooks/plex-remote-access.md diff --git a/apps/wave-0/kustomization.yaml b/apps/wave-0/kustomization.yaml index 50b5244..49a173a 100644 --- a/apps/wave-0/kustomization.yaml +++ b/apps/wave-0/kustomization.yaml @@ -18,6 +18,7 @@ resources: - pihole-oauth2-proxy-credentials.sops.yaml - pihole.yaml - plex-claim-secret.sops.yaml +- plex-remote.yaml - plex.yaml - qbittorrent-bootstrap-secret.sops.yaml - qbittorrent-oauth2-proxy-credentials.sops.yaml diff --git a/apps/wave-0/plex-remote.yaml b/apps/wave-0/plex-remote.yaml new file mode 100644 index 0000000..595cc4c --- /dev/null +++ b/apps/wave-0/plex-remote.yaml @@ -0,0 +1,126 @@ +--- +# ================================================================================================= +# PLEX REMOTE ACCESS — raw tcp/32400 north-south exposure, and the policy that admits it. +# ================================================================================================= +# HAND-WRITTEN, NOT RENDERED. A non-catalog app-tier unit anchored to the `plex` ServiceDefinition +# and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS table — the same +# mechanism gitea-actions and pihole-dns use. It is hand-written for the SAME reason those are: +# schemas/servicedefinition.schema.json is `additionalProperties: false` and has NO field for a +# type=LoadBalancer Service, for a pinned lbipam VIP, or for raw-TCP north-south exposure at all. +# `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator. +# +# WHY THE L7 GATEWAY IS NOT ENOUGH HERE — and it is NOT because the Gateway fails to serve Plex. +# Measured 2026-10-03 through the Gateway VIP: `GET https://plex.home.keithjackson.dev/identity` +# returned HTTP 200 with `claimed="1"` and the real machineIdentifier, and the protected endpoints +# (/library/sections, /status/sessions, /:/prefs) all returned 401. The L7 path WORKS. +# +# The problem is DISCOVERY, not proxying. Plex apps (phone, TV, Roku) do not resolve +# `plex.${BASE_DOMAIN}`; they ask plex.tv for the server's connection list. A server gets onto that +# list one of two ways: it advertises a custom URL (`customConnections`, the ADVERTISE_IP env), or it +# publishes a reachable public IP:port for its own tcp/32400. Measured in the live Preferences.xml on +# 2026-10-03: `customConnections` ABSENT, `ManualPortMappingMode` ABSENT, while +# `PublishServerOnPlexOnlineKey="1"` — the server was trying to publish and had nothing reachable to +# publish, so remote clients saw NO connection at all. The operator chose the NATIVE path (this +# file) over advertising the Gateway URL: Plex then terminates its own TLS with its +# `*.plex.direct` certificate and video streams bypass the Envoy L7 hop entirely. +# +# VIP: ${PLEX_L4_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.57, prod 192.168.2.24), PINNED via +# lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool` +# (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the existing +# CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config is needed; +# this Service just consumes the pool. PINNING IS NOT OPTIONAL: the address is typed into a WAN +# port-forward on the SOHO router, which is hand-maintained and off-repo, so it can never move. +# 192.168.2.24 verified FREE live 2026-10-03 — `default-pool` reported IPsTotal 11 / IPsUsed 4 / +# IPsAvailable 7, and the only four allocations were .20 (cilium-gateway-shared), .21 (unifi-l4-*), +# .22 (crafty-l4-*) and .23 (pihole-dns-*). +# +# ONE SERVICE, TCP ONLY. Unlike unifi/crafty/pihole-dns this needs no second Service and no +# lbipam.cilium.io/sharing-key: Plex's remote access is tcp/32400 and nothing else. (Plex's +# DLNA/GDM discovery uses UDP, but that is LAN-only multicast and is not part of remote access.) +# +# NO `loadBalancerSourceRanges`, DELIBERATELY — and this is the one place this file diverges from +# its closest sibling. apps/wave-0/pihole-dns.yaml pins its source range to ${LAN_CIDR} because a +# resolver must never answer the internet. This Service is the OPPOSITE case: admitting remote +# clients from arbitrary internet addresses is its entire purpose, so there is no range to narrow +# to. Plex authenticates every caller itself (measured above: 401 on every protected endpoint +# without a token), which is what makes an unfiltered port defensible here. +# +# NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry +# `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their VIP as a public +# Cloudflare record. This Service must not, for pihole-dns's reason: ${PLEX_L4_VIP} is an RFC1918 +# address that is useless in public DNS. Remote clients get this server's address from plex.tv, not +# from the zone. The existing public `plex.${BASE_DOMAIN}` record is published from the rendered +# HTTPRoute via the gateway source and is NOT affected by this file — the web UI keeps working +# through the Gateway exactly as before. +# +# ------------------------------------------------------------------------------------------------- +# externalTrafficPolicy: Cluster — inherited from crafty's 8-DAY LAN OUTAGE, not re-derived. +# ------------------------------------------------------------------------------------------------- +# `Local` is the tempting choice because it preserves the client source IP, which this service +# actually has a use for (see the allowedNetworks note below). It is still WRONG here, for the +# reason platform/crafty/service-l4.yaml documents at length: Cilium's L2 announcer elects ONE lease +# holder per Service and does NOT follow the backend pod, so with `Local` the VIP can be announced +# by a node that has no Plex pod, which black-holes the port while every object still reports +# healthy. Plex is a single-replica Deployment with `strategy: Recreate` pinned to wherever the +# scheduler put it, so the mismatch is the normal case, not a rare one. Take the SNAT. +# +# KNOWN CONSEQUENCE OF THAT CHOICE, stated rather than discovered later: `Cluster` SNATs the client +# to the ingress NODE's address (192.168.2.1x), which falls inside the `192.168.0.0/16` entry of +# catalog/plex.yaml's ALLOWED_NETWORKS. Plex will therefore classify every REMOTE client as LAN. +# Measured 2026-10-03 on the Gateway path, which SNATs the same way: Plex logged the request as +# `[10.0.2.214:36040 (Allowed Network (WAN))]`. This is a METERING effect, not a security one — +# authorization is unchanged (401s above), but Plex's remote-quality/bandwidth limits will not bind +# on these streams because it believes they are local. Narrowing ALLOWED_NETWORKS cannot fix it +# without also de-classifying genuine LAN clients, so it is accepted. +apiVersion: v1 +kind: Service +metadata: + name: plex-l4-tcp + namespace: plex + labels: + app.kubernetes.io/name: plex + annotations: + lbipam.cilium.io/ips: "${PLEX_L4_VIP}" +spec: + type: LoadBalancer + externalTrafficPolicy: Cluster + selector: + app.kubernetes.io/name: plex + ports: + - name: plex-remote + port: 32400 + targetPort: 32400 + protocol: TCP +--- +# INGRESS allow for the raw port (R-NET-01/02). +# +# The rendered plex.yaml already carries `plex-allow-gateway-ingress` (fromEntities: ingress -> +# 32400) and three reciprocal allows for sonarr/radarr/seerr. CiliumNetworkPolicies are ADDITIVE, so +# this file adds the off-cluster case without touching those. +# +# `world` is Cilium's entity for anything outside the cluster, which is what a remote Plex client is. +# `host` and `cluster` mirror platform/crafty/networkpolicy.yaml's game-port rule and are what make +# this work under `externalTrafficPolicy: Cluster`: after SNAT the packet's source is the ingress +# NODE, which Cilium classifies as `host` (same node) or `remote-node` (a different one) — NOT +# `world`. Listing `world` alone would compile cleanly and then drop every remote stream, so all +# four entities are named. +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: plex-allow-remote-32400 + namespace: plex + labels: + app.kubernetes.io/name: plex +spec: + endpointSelector: + matchLabels: + app.kubernetes.io/name: plex + ingress: + - fromEntities: + - world + - cluster + - host + - remote-node + toPorts: + - ports: + - { port: "32400", protocol: TCP } diff --git a/catalog/render.py b/catalog/render.py index 24d1950..0b27f8b 100644 --- a/catalog/render.py +++ b/catalog/render.py @@ -2002,6 +2002,24 @@ EXTRA_UNITS: dict = { "pihole-dns.yaml", ], }, + # Plex's remote-access exposure: the raw tcp/32400 LoadBalancer Service on a pinned VIP, plus the + # CiliumNetworkPolicy that admits off-cluster clients to it (apps/wave-0/plex-remote.yaml). + # `above: False` co-locates it in plex's OWN wave — these are the plex pod's own Service and its + # own policy, applied in the same wave as the Deployment they select, exactly as the rendered + # Service and CNPs in plex.yaml are. Nothing in a higher wave gates on it. + # + # It is a non-catalog unit for the same reason pihole-dns is: the ServiceDefinition schema cannot + # express a LoadBalancer Service or a pinned lbipam VIP (additionalProperties is false, and + # `ingressRoutes` only renders HTTPRoutes on the L7 Gateway, which cannot carry Plex's raw port). + # Plex's own settings stay in the catalog as env on catalog/plex.yaml; the only thing living + # outside it is the exposure the schema has no word for. + "plex-remote": { + "anchor": "plex", + "above": False, + "files": [ + "plex-remote.yaml", + ], + }, } diff --git a/clusters/base/cluster-vars.yaml b/clusters/base/cluster-vars.yaml index 802443e..026ee84 100644 --- a/clusters/base/cluster-vars.yaml +++ b/clusters/base/cluster-vars.yaml @@ -100,7 +100,8 @@ data: # platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY # tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db: # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus - # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers), + # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers; + # plex-remote: PLEX_L4_VIP), # velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION, # VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an # UNSUBSTITUTED path would ship the literal string into the cluster. @@ -175,6 +176,19 @@ data: # dev -> 192.168.1.56 (base default; next free after crafty .55 in dev's .53-.60 pool) # prod -> 192.168.2.23 (next free after crafty .22 in prod's .20-.30 pool; see clusters/prod) PIHOLE_DNS_VIP: "192.168.1.56" # PiHole: authoritative LAN DNS for ${BASE_DOMAIN} + # Plex's remote-access listener: raw tcp/32400 for REMOTE Plex clients (apps/wave-0/plex-remote.yaml). + # A FOURTH pinned L4 VIP, for the usual reason — the `shared` Gateway is an L7 HTTPS terminator and + # cannot carry Plex's own TLS. Plex clients do not resolve `plex.${BASE_DOMAIN}`; they fetch this + # server's address from plex.tv, so the Gateway's L7 path (which still serves the web UI, and was + # measured working) cannot put the server on a client's connection list at all. + # + # PINNED, AND THE PIN IS LOAD-BEARING OFF-REPO: the SOHO router forwards WAN tcp/32400 to this + # address by hand. A VIP taken from the pool head would silently move on a reschedule and break + # remote access with nothing in-cluster looking wrong. See runbooks/plex-remote-access.md. + # + # dev -> 192.168.1.57 (base default; next free after pihole-dns .56 in dev's .53-.60 pool) + # prod -> 192.168.2.24 (next free after pihole-dns .23 in prod's .20-.30 pool; see clusters/prod) + PLEX_L4_VIP: "192.168.1.57" # Plex: tcp/32400 remote access (NOT the web UI, which is L7) # The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress # peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable diff --git a/clusters/prod/kustomization.yaml b/clusters/prod/kustomization.yaml index ec00743..e518e64 100644 --- a/clusters/prod/kustomization.yaml +++ b/clusters/prod/kustomization.yaml @@ -476,6 +476,26 @@ patches: - op: replace path: /data/PIHOLE_DNS_VIP value: "192.168.2.23" + # Plex's remote-access listener VIP — raw tcp/32400 for REMOTE Plex clients + # (apps/wave-0/plex-remote.yaml). 192.168.2.24 is the NEXT FREE address in the prod pool + # 192.168.2.20-.30, verified live 2026-10-03: the CiliumLoadBalancerIPPool `default-pool` + # reported IPsTotal 11 / IPsUsed 4 / IPsAvailable 7, and the only four allocations are .20 + # (cilium-gateway-shared), .21 (unifi-l4-tcp/-udp), .22 (crafty-l4-tcp/-udp) and .23 + # (pihole-dns-tcp/-udp). It is inside the pool block the operator already asserted sits outside + # the SOHO router's DHCP scope, and it is not a node address (.10/.11/.12) or the + # control-plane VIP (.9). + # + # UNLIKE .23, THIS ONE IS ALSO A WAN PORT-FORWARD TARGET. The router forwards WAN tcp/32400 + # here, by hand, off-repo — so changing this value is a two-place edit and the router is the + # half that CI cannot check. runbooks/plex-remote-access.md is the procedure. + # + # NOT ARP-PROBED. .23 was verified free with `arping` from a LAN host before it was claimed; + # for .24 only the in-cluster pool accounting above was checked, so an off-cluster squatter on + # this address would not have been caught. Probe before trusting it if remote access comes up + # with a conflict. + - op: replace + path: /data/PLEX_L4_VIP + value: "192.168.2.24" # The prod flat LAN. This is the 192.168.2.0/24 segment the SOHO ROUTER at 192.168.2.1 owns — # NOT pfSense's 192.168.1.0/24, which is a different network and not in the LAN-DNS path. # It scopes the ingress allow for PiHole's DNS listener to LAN clients only. diff --git a/decisions/2026-10-03-plex-remote-access-l4.md b/decisions/2026-10-03-plex-remote-access-l4.md new file mode 100644 index 0000000..c26d958 --- /dev/null +++ b/decisions/2026-10-03-plex-remote-access-l4.md @@ -0,0 +1,129 @@ +--- +date: 2026-10-03 +branch: feat/plex-remote-access-l4 +title: Plex remote access — raw tcp/32400 via a pinned LoadBalancer VIP, not the L7 Gateway +--- + +## What changed + +Reported symptom: users cannot reach Plex from outside the LAN. **Two independent causes**, measured +rather than inferred: + +1. **Plex advertised no remote connection at all.** Live `Preferences.xml` on prod: + `customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while + `PublishServerOnPlexOnlineKey="1"` — publishing was on and there was nothing reachable to publish. + Plex's automatic UPnP/NAT-PMP path cannot work from a pod on the Cilium overlay. +2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` refused; + `136.51.115.252:443` answered with a **self-signed router cert** (`C=ZZ, CN=%D`), not the + Gateway's LE wildcard. The Plex Service was `ClusterIP` only — no off-cluster path to 32400 by + any route. + +This branch fixes the in-cluster half. **Remote access does not work until the operator also does +two manual steps** (`runbooks/plex-remote-access.md`): the WAN tcp/32400 port-forward, and ticking +"manually specify public port" in the Plex UI. + +- **`apps/wave-0/plex-remote.yaml` (new, hand-written).** A `type=LoadBalancer` Service on tcp/32400 + pinned to `${PLEX_L4_VIP}` via `lbipam.cilium.io/ips`, plus a `CiliumNetworkPolicy` + (`plex-allow-remote-32400`) admitting `world`/`cluster`/`host`/`remote-node` to 32400. Non-catalog + because `schemas/servicedefinition.schema.json` is `additionalProperties: false` and has no field + for a LoadBalancer Service or a pinned VIP — the same reason `apps/wave-0/pihole-dns.yaml`, + `platform/crafty/service-l4.yaml` and `apps/vms/unifi/50-service-l4.yaml` are hand-written. +- **`catalog/render.py`:** an `EXTRA_UNITS` entry (`plex-remote`, anchored to `plex`, `above: False`) + so `--emit-waves` places the unit in plex's own wave. +- **`clusters/base/cluster-vars.yaml`:** new `PLEX_L4_VIP` (dev default `192.168.1.57`), plus the + substitution inventory comment updated to name `plex-remote: PLEX_L4_VIP`. +- **`clusters/prod/kustomization.yaml`:** prod override `PLEX_L4_VIP: 192.168.2.24`. +- **`apps/wave-0/kustomization.yaml`:** `plex-remote.yaml` added to `resources`. +- **`runbooks/plex-remote-access.md` (new):** the two manual steps, the four-step verification, why + no alert is possible, and the two documented warts below. + +**`ADVERTISE_IP` was deliberately NOT set.** Setting it to `https://plex.home.keithjackson.dev:443` +was the alternative — it needs no new WAN hole and the mechanism was verified working on an +already-claimed server. **The operator chose the native path (2026-10-03)** so Plex terminates its own +TLS (`*.plex.direct`) and streams bypass the Envoy L7 hop. `catalog/plex.yaml`'s deferral note for +`ADVERTISE_IP` remains accurate and untouched. + +**`externalTrafficPolicy: Cluster`, with a known cost.** `Local` would preserve the client IP, but +Cilium's L2 announcer does not follow the backend pod — the wart that cost this LAN an 8-day +Minecraft outage (see `platform/crafty/service-l4.yaml`). Consequence accepted and documented: SNAT +puts the client in `192.168.0.0/16`, inside `ALLOWED_NETWORKS`, so **Plex classifies remote streams +as LAN**. That is a metering effect (remote bandwidth limits will not bind), not an authorization one. + +**SENSITIVE PATHS — this PR touches `clusters/**` (two files), so `ci/approval-policy.sh` will BLOCK +it.** It needs the operator's `/approve-override ` posted as a PR **review** of type +"Comment". I have deliberately not posted it: CLAUDE.md §1 forbids an agent clearing this on its own +PR, and the agent account holds `owner`, so the prohibition is a rule, not a limit. + +## What I verified + +All measurements against live prod, 2026-10-03, read-only. + +- **Plex's advertised state**, the finding this branch rests on: `customConnections=`, + `ManualPortMappingMode=`, `ManualPortMappingPort=`, + `PublishServerOnPlexOnlineKey="1"`, `allowedNetworks="10.0.0.0/8,192.168.0.0/16"`. Read by + attribute grep, never by dumping `Preferences.xml` (it holds `PlexOnlineToken`). +- **The L7 Gateway path works and is NOT the problem.** Via the Gateway VIP, + `GET https://plex.home.keithjackson.dev/identity` → `HTTP 200`, `claimed="1"`, real + `machineIdentifier`. HTTPRoute `plex-32400`: `Accepted=True`, `ResolvedRefs=True`. +- **Plex enforces its own auth**, which is what makes an unfiltered WAN port defensible: + `/library/sections`, `/status/sessions`, `/:/prefs` all → **401** unauthenticated through the + Gateway. +- **The SNAT/allowedNetworks wart is real, not theoretical.** A tagged probe through the Gateway was + logged by Plex as `[10.0.2.214:36040 (Allowed Network (WAN))]` — the Envoy pod IP, classified as an + allowed network. +- **WAN state:** `136.51.115.252:32400` → connection refused. `136.51.115.252:443` → self-signed + cert `C=ZZ, ST=Somewhere, O=%D342e7e6b, CN=%D`. Public DNS resolves correctly: + `plex.home.keithjackson.dev` → CNAME `wan.home.keithjackson.dev` → `136.51.115.252`. +- **`192.168.2.24` is free in-cluster.** `CiliumLoadBalancerIPPool default-pool`: IPsTotal 11 / + IPsUsed 4 / IPsAvailable 7; the only allocations are `.20` (cilium-gateway-shared), `.21` + (unifi-l4-*), `.22` (crafty-l4-*), `.23` (pihole-dns-*). Not a node address (`.10/.11/.12`) and not + the kube-vip control-plane VIP (`.9`). +- **The manual-port-mapping step genuinely cannot be automated.** Grep of the running image's whole + init surface (`/etc/cont-init.d`, `/plex-common.sh`, `/usr/local/bin`) for `ManualPortMapping` and + `PLEX_PREFERENCE` → **NONE**. +- **`ADVERTISE_IP` would have worked on a claimed server** (recorded because the runbook's + "why not" rests on it): `setPref "customConnections"` at line 126 of `40-plex-first-run` sits + **outside** the `PLEX_CLAIM`/`PlexOnlineToken` guard, and the `/.firstRunComplete` early-exit + marker is on the **overlay** FS, not the `/config` PVC, so the script re-runs every pod start. +- **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with + `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all + **pass**. The rc 3 is two environmental skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`) + and `conformance` L2/L3 (`CLUSTER_AVAILABLE` unset). **Not green. Reported as rc 3, not as pass.** +- **`render-golden` passing is the specific proof** that the `render.py` `EXTRA_UNITS` change, the new + hand-written unit and the edited wave kustomization are mutually consistent. + +## What I did NOT verify + +- **THE ACTUAL FIX. Nothing in this branch has been proven to make remote access work** — it cannot + be, because both remaining steps are off-repo and manual (router forward, Plex UI toggle). This + branch removes two blockers and adds none; it does not demonstrate an outcome. +- **Outside-in reachability, at all.** Every probe above was taken from inside the LAN, so the + `:443` self-signed cert strictly proves only that there is **no NAT hairpin** — it does **not** + prove outside-in failure. A router cert on WAN:443 is suggestive (its management UI would shadow a + forward), but unproven. **This repo has no off-LAN vantage point.** Test 3 in the runbook + (app.plex.tv from a cellular connection) is the only honest check. +- **`192.168.2.24` was never ARP-probed off-cluster.** Only in-cluster pool accounting was checked. + `.23` was `arping`-probed before it was claimed; `.24` was not, so an off-cluster squatter would + not have been caught. +- **The LoadBalancer has never been reconciled.** `plex-l4-tcp` does not exist in prod yet, so + LB-IPAM has not actually assigned the pinned VIP, L2 announcement has not been observed for it, and + the new CiliumNetworkPolicy has never admitted a real packet. Whether `world`/`host`/`cluster`/ + `remote-node` is the correct entity set **under this SNAT path** is reasoned from + `platform/crafty/networkpolicy.yaml`'s working game-port rule, **not measured**. +- **Whether Plex's remote path needs anything beyond the manual port tick.** If plex.tv still lists + no remote connection after both manual steps, this branch is necessary-but-insufficient. +- **The CI approval-policy outcome.** The local stage reported green, but that is an offline N/A, not + a verdict: this PR touches `clusters/**`, so CI will block it pending an override. +- **PRE-EXISTING DRIFT I FOUND AND DELIBERATELY LEFT ALONE — worth its own follow-up.** Running the + documented `python3 catalog/render.py --emit-waves` regenerates the wave kustomizations and wants to + add **`vm-demo.yaml`** to wave 0 and **`keycloak.yaml`** to wave 1. Both are listed as + intentionally not onboarded in `ci/render-golden.sh` + (`RENDER_GOLDEN_NOT_ONBOARDED="keycloak vm-demo"`); their manifests are tracked in `apps/` but + deliberately absent from the `resources` lists so Flux does not apply them. `--emit-waves` places + any file present on disk and has **no awareness of that exclusion**, while `render-golden.sh` uses + `--wave-layout` and compares rendered manifests only — it never checks kustomization `resources`, + which is why this has survived unnoticed. **So the header "DO NOT hand-edit: re-run + `render.py --emit-waves`" is currently unsafe advice**: following it onboards an IdP and a KubeVirt + VM. I therefore restored both kustomizations and hand-added only my one line. I did not fix + `--emit-waves`, and I have not determined how long the drift has existed or whether either service + is *meant* to be onboarded. diff --git a/runbooks/plex-remote-access.md b/runbooks/plex-remote-access.md new file mode 100644 index 0000000..dad33e7 --- /dev/null +++ b/runbooks/plex-remote-access.md @@ -0,0 +1,148 @@ +# plex-remote-access — getting Plex reachable from outside the LAN, and proving it + +**Status: the in-cluster half is shipped; the last mile is TWO MANUAL STEPS and remote access does +NOT work until both are done.** Neither can live in this repo. +**Alert:** none, and none is possible from in-cluster signals — see "Why no alert" below. This is an +**operator procedure**, not an incident page. +**Files:** `apps/wave-0/plex-remote.yaml` (the tcp/32400 LoadBalancer + its CiliumNetworkPolicy), +`clusters/base/cluster-vars.yaml` + `clusters/prod/kustomization.yaml` (`PLEX_L4_VIP`), +`catalog/render.py` (the `EXTRA_UNITS` entry that places the unit), `catalog/plex.yaml` (Plex's own +env — note `ADVERTISE_IP` is deliberately NOT set; see "Why not the Gateway URL"). +**Related:** `runbooks/plex-quicksync.md` and `runbooks/plex-claim.md` — the other two Plex +procedures with a manual last mile. + +## What was actually broken (measured 2026-10-03) + +Plex was serving correctly and was reachable on the LAN. Two independent things stopped remote +clients: + +1. **Plex advertised no remote connection at all.** In the live `Preferences.xml`: + `customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while + `PublishServerOnPlexOnlineKey="1"`. The server was trying to publish itself to plex.tv and had + nothing reachable to publish. Plex's automatic path (UPnP/NAT-PMP) cannot work here: the process + lives in a pod on the Cilium overlay (`10.0.2.136`), it cannot speak UPnP to the SOHO router, and + a mapping to the pod IP would be meaningless if it could. +2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` → connection refused. + `136.51.115.252:443` → a **self-signed router certificate** (`C=ZZ, CN=%D`), not the Gateway's + Let's Encrypt wildcard. + +The Service was `ClusterIP` only, so port 32400 had no path off-cluster by any route. + +**Caveat on that second measurement, because it is easy to over-read:** it was taken from *inside* +the LAN, so strictly it proves there is no NAT hairpin (reflection). It does not prove outside-in +failure. But a router certificate answering 443 means the router's own management UI is bound to the +WAN port, and that *would* shadow a port-forward for genuine outside traffic too. **Verify from a +real outside vantage point** (phone on cellular, or an external port checker) before concluding +either way — this repo has no such vantage point and must not pretend otherwise. + +## Step 1 — forward WAN tcp/32400 on the router (operator, off-repo) + + WAN tcp/32400 -> 192.168.2.24 (${PLEX_L4_VIP}) port 32400 + +`192.168.2.24` is a pinned Cilium LB VIP, ARP-announced on the flat LAN by the existing +`CiliumL2AnnouncementPolicy default-l2`. It is **pinned on purpose**: this address is typed into the +router by hand, so it must never move on a reschedule. + +While in the router, **check what else is bound to the WAN interface.** If remote management is +listening on 443 (see the certificate measurement above), move it or disable it — otherwise the +Gateway's 80/443 forward is shadowed and the Plex *web UI* over the internet stays broken even +though remote *streaming* works. These are two separate paths and they fail independently. + +Confirm the VIP is actually answering on the LAN first, so a failure later is unambiguous: + + ./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n plex get svc plex-l4-tcp -o wide' + +`EXTERNAL-IP` must read `192.168.2.24`. If it is ``, LB-IPAM could not assign — the address +is outside the pool or already taken; see `clusters/prod/kustomization.yaml` for the pool accounting. + +**`192.168.2.24` was never ARP-probed off-cluster.** Only the in-cluster pool accounting was checked +(IPsTotal 11 / IPsUsed 4 / IPsAvailable 7 on 2026-10-03). If the VIP behaves oddly, probe it from a +LAN host — `arping 192.168.2.24` — the way `.23` was probed before it was claimed. + +## Step 2 — tick "manually specify public port" in Plex (operator, UI only) + +**There is no GitOps path to this and no environment variable for it.** Measured in the running +image on 2026-10-03: a grep of the entire init surface (`/etc/cont-init.d`, `/plex-common.sh`, +`/usr/local/bin`) for `ManualPortMapping` and `PLEX_PREFERENCE` returned **NONE**. The image writes +only six preferences, and this is not one of them. It is the same class of manual last mile as the +`HardwareAcceleratedCodecs` toggle in `runbooks/plex-quicksync.md`. + +In the Plex web UI (`https://plex.home.keithjackson.dev`, on the LAN): + +**Settings → Remote Access** +- Ensure **Enable Remote Access** is on. (`PublishServerOnPlexOnlineKey` was already `1`.) +- Tick **Manually specify public port** and set it to **32400**. +- **Apply**, then let the panel settle. It should report *"Fully accessible outside your network"*. + +Without this tick Plex waits for a NAT mapping it can never obtain and publishes no remote +connection, so the port-forward in step 1 sits there unused. + +## How to verify + +Do these in order; each one isolates a different half. + +**1. The port is open from outside.** From a phone on cellular, or any off-LAN host: + + curl -sk -o /dev/null -w '%{http_code}\n' https://136.51.115.252:32400/identity + +`200` means the forward and the policy are both good. A hang or refusal means step 1. + +**2. Plex itself believes it is reachable.** Settings → Remote Access must say *"Fully accessible +outside your network"*. If it says *"not available outside your network"* while test 1 passes, step 2 +was not applied. + +**3. plex.tv is handing out a remote connection.** The decisive test, because it is what clients +actually consume: sign in to [app.plex.tv](https://app.plex.tv) from an off-LAN network and confirm +the server appears and plays. A client finds this server through plex.tv's connection list, never +through DNS. + +**4. The policy is admitting the traffic.** If test 1 fails *after* the forward is confirmed, the +CiliumNetworkPolicy is the next suspect: + + ./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n plex get cnp plex-allow-remote-32400' + +See `runbooks/denied-flows.md` for reading actual drops. + +## Why no alert + +Every in-cluster signal is green whether or not remote access works. The Service has endpoints, the +pod is `Running`, the policy is applied, and `plex-down` watches +`kube_deployment_status_replicas_available` — none of which can see a router with a missing +port-forward or an unticked checkbox. An alert would need an off-LAN prober, which this cluster does +not have. **Test 3 above is the only honest check, and it is manual.** + +## Why not the Gateway URL (`ADVERTISE_IP`) + +The alternative was to leave Plex on `ClusterIP` and set +`ADVERTISE_IP: https://plex.home.keithjackson.dev:443`, which writes Plex's `customConnections` +preference and would have put the server on plex.tv's list via the existing L7 Gateway — no new WAN +hole. That mechanism was verified to work on an already-claimed server (the `ADVERTISE_IP` block at +line 126 of the image's `40-plex-first-run` sits **outside** the claim guard, and `/.firstRunComplete` +lives on the overlay FS, not the `/config` PVC, so it re-runs every pod start), and the Gateway path +itself was measured serving Plex correctly: `HTTP 200`, `claimed="1"`, with `401` on every protected +endpoint. + +**The operator chose the native path anyway (2026-10-03), and the reasons are real:** Plex terminates +its own TLS with its `*.plex.direct` certificate, and video streams bypass the Envoy L7 hop instead +of being proxied through it. The cost is a new unfiltered WAN port — acceptable because Plex +authenticates every caller itself. `ADVERTISE_IP` therefore stays unset in `catalog/plex.yaml`, and +the deferral note there is still accurate. + +## Known wart: remote clients are classified as LAN + +`apps/wave-0/plex-remote.yaml` uses `externalTrafficPolicy: Cluster`, which SNATs the client to the +ingress node's address (`192.168.2.1x`). That falls inside the `192.168.0.0/16` entry of +`ALLOWED_NETWORKS`, so **Plex treats every remote stream as local.** Measured on the Gateway path, +which SNATs the same way: Plex logged `[10.0.2.214:36040 (Allowed Network (WAN))]`. + +This is a **metering** effect, not a security one — authorization is unchanged, as the 401s above +show — but Plex's remote-quality and bandwidth limits will not bind on these streams. + +`Local` would preserve the client IP and fix it. **Do not switch to it.** Cilium's L2 announcer +elects one lease holder per Service and does *not* follow the backend pod, so the VIP can be +announced by a node with no Plex pod, black-holing the port while everything reports healthy. That +exact change cost this LAN an 8-day Minecraft outage in September 2026 — the full write-up is at the +top of `platform/crafty/service-l4.yaml`. Plex is a single-replica `Recreate` Deployment, so the +node mismatch is the normal case here, not a rare one. -- 2.54.0