home-network-k3s-impl/decisions/2026-10-03-plex-remote-access-l4.md

date, branch, title

date	branch	title
2026-10-03	feat/plex-remote-access-l4	Plex remote access — raw tcp/32400 via a pinned LoadBalancer VIP, not the L7 Gateway

What changed

Reported symptom: users cannot reach Plex from outside the LAN. Two independent causes, measured rather than inferred:

1. Plex advertised no remote connection at all. Live Preferences.xml on prod: customConnections ABSENT, ManualPortMappingMode ABSENT, while PublishServerOnPlexOnlineKey="1" — publishing was on and there was nothing reachable to publish. Plex's automatic UPnP/NAT-PMP path cannot work from a pod on the Cilium overlay.
2. Nothing from the WAN reached the cluster. 136.51.115.252:32400 refused; 136.51.115.252:443 answered with a self-signed router cert (C=ZZ, CN=%D), not the Gateway's LE wildcard. The Plex Service was ClusterIP only — no off-cluster path to 32400 by any route.

This branch fixes the in-cluster half. Remote access does not work until the operator also does two manual steps (runbooks/plex-remote-access.md): the WAN tcp/32400 port-forward, and ticking "manually specify public port" in the Plex UI.

• apps/wave-0/plex-remote.yaml (new, hand-written). A type=LoadBalancer Service on tcp/32400 pinned to ${PLEX_L4_VIP} via lbipam.cilium.io/ips, plus a CiliumNetworkPolicy (plex-allow-remote-32400) admitting world/cluster/host/remote-node to 32400. Non-catalog because schemas/servicedefinition.schema.json is additionalProperties: false and has no field for a LoadBalancer Service or a pinned VIP — the same reason apps/wave-0/pihole-dns.yaml, platform/crafty/service-l4.yaml and apps/vms/unifi/50-service-l4.yaml are hand-written.
• catalog/render.py: an EXTRA_UNITS entry (plex-remote, anchored to plex, above: False) so --emit-waves places the unit in plex's own wave.
• clusters/base/cluster-vars.yaml: new PLEX_L4_VIP (dev default 192.168.1.57), plus the substitution inventory comment updated to name plex-remote: PLEX_L4_VIP.
• clusters/prod/kustomization.yaml: prod override PLEX_L4_VIP: 192.168.2.24.
• apps/wave-0/kustomization.yaml: plex-remote.yaml added to resources.
• runbooks/plex-remote-access.md (new): the two manual steps, the four-step verification, why no alert is possible, and the two documented warts below.

ADVERTISE_IP was deliberately NOT set. Setting it to https://plex.home.keithjackson.dev:443 was the alternative — it needs no new WAN hole and the mechanism was verified working on an already-claimed server. The operator chose the native path (2026-10-03) so Plex terminates its own TLS (*.plex.direct) and streams bypass the Envoy L7 hop. catalog/plex.yaml's deferral note for ADVERTISE_IP remains accurate and untouched.

externalTrafficPolicy: Cluster, with a known cost. Local would preserve the client IP, but Cilium's L2 announcer does not follow the backend pod — the wart that cost this LAN an 8-day Minecraft outage (see platform/crafty/service-l4.yaml). Consequence accepted and documented: SNAT puts the client in 192.168.0.0/16, inside ALLOWED_NETWORKS, so Plex classifies remote streams as LAN. That is a metering effect (remote bandwidth limits will not bind), not an authorization one.

SENSITIVE PATHS — this PR touches clusters/** (two files), so ci/approval-policy.sh will BLOCK it. It needs the operator's /approve-override <reason> posted as a PR review of type "Comment". I have deliberately not posted it: CLAUDE.md §1 forbids an agent clearing this on its own PR, and the agent account holds owner, so the prohibition is a rule, not a limit.

What I verified

All measurements against live prod, 2026-10-03, read-only.

• Plex's advertised state, the finding this branch rests on: customConnections=<ABSENT>, ManualPortMappingMode=<ABSENT>, ManualPortMappingPort=<ABSENT>, PublishServerOnPlexOnlineKey="1", allowedNetworks="10.0.0.0/8,192.168.0.0/16". Read by attribute grep, never by dumping Preferences.xml (it holds PlexOnlineToken).
• The L7 Gateway path works and is NOT the problem. Via the Gateway VIP, GET https://plex.home.keithjackson.dev/identity → HTTP 200, claimed="1", real machineIdentifier. HTTPRoute plex-32400: Accepted=True, ResolvedRefs=True.
• Plex enforces its own auth, which is what makes an unfiltered WAN port defensible: /library/sections, /status/sessions, /:/prefs all → 401 unauthenticated through the Gateway.
• The SNAT/allowedNetworks wart is real, not theoretical. A tagged probe through the Gateway was logged by Plex as [10.0.2.214:36040 (Allowed Network (WAN))] — the Envoy pod IP, classified as an allowed network.
• WAN state: 136.51.115.252:32400 → connection refused. 136.51.115.252:443 → self-signed cert C=ZZ, ST=Somewhere, O=%D342e7e6b, CN=%D. Public DNS resolves correctly: plex.home.keithjackson.dev → CNAME wan.home.keithjackson.dev → 136.51.115.252.
• 192.168.2.24 is free in-cluster. CiliumLoadBalancerIPPool default-pool: IPsTotal 11 / IPsUsed 4 / IPsAvailable 7; the only allocations are .20 (cilium-gateway-shared), .21 (unifi-l4-), .22 (crafty-l4-), .23 (pihole-dns-*). Not a node address (.10/.11/.12) and not the kube-vip control-plane VIP (.9).
• The manual-port-mapping step genuinely cannot be automated. Grep of the running image's whole init surface (/etc/cont-init.d, /plex-common.sh, /usr/local/bin) for ManualPortMapping and PLEX_PREFERENCE → NONE.
• ADVERTISE_IP would have worked on a claimed server (recorded because the runbook's "why not" rests on it): setPref "customConnections" at line 126 of 40-plex-first-run sits outside the PLEX_CLAIM/PlexOnlineToken guard, and the /.firstRunComplete early-exit marker is on the overlay FS, not the /config PVC, so the script re-runs every pod start.
• Toolbox gate (./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'): rc 3, with spec-immutability, decisions-log, lint, l0, render-golden, l1 and approval-policy all pass. The rc 3 is two environmental skips — gitea-settings (no local SETTINGS_API_TOKEN) and conformance L2/L3 (CLUSTER_AVAILABLE unset). Not green. Reported as rc 3, not as pass.
• render-golden passing is the specific proof that the render.py EXTRA_UNITS change, the new hand-written unit and the edited wave kustomization are mutually consistent.

What I did NOT verify

• THE ACTUAL FIX. Nothing in this branch has been proven to make remote access work — it cannot be, because both remaining steps are off-repo and manual (router forward, Plex UI toggle). This branch removes two blockers and adds none; it does not demonstrate an outcome.
• Outside-in reachability, at all. Every probe above was taken from inside the LAN, so the :443 self-signed cert strictly proves only that there is no NAT hairpin — it does not prove outside-in failure. A router cert on WAN:443 is suggestive (its management UI would shadow a forward), but unproven. This repo has no off-LAN vantage point. Test 3 in the runbook (app.plex.tv from a cellular connection) is the only honest check.
• 192.168.2.24 was never ARP-probed off-cluster. Only in-cluster pool accounting was checked. .23 was arping-probed before it was claimed; .24 was not, so an off-cluster squatter would not have been caught.
• The LoadBalancer has never been reconciled. plex-l4-tcp does not exist in prod yet, so LB-IPAM has not actually assigned the pinned VIP, L2 announcement has not been observed for it, and the new CiliumNetworkPolicy has never admitted a real packet. Whether world/host/cluster/ remote-node is the correct entity set under this SNAT path is reasoned from platform/crafty/networkpolicy.yaml's working game-port rule, not measured.
• Whether Plex's remote path needs anything beyond the manual port tick. If plex.tv still lists no remote connection after both manual steps, this branch is necessary-but-insufficient.
• The CI approval-policy outcome. The local stage reported green, but that is an offline N/A, not a verdict: this PR touches clusters/**, so CI will block it pending an override.
• PRE-EXISTING DRIFT I FOUND AND DELIBERATELY LEFT ALONE — worth its own follow-up. Running the documented python3 catalog/render.py --emit-waves regenerates the wave kustomizations and wants to add vm-demo.yaml to wave 0 and keycloak.yaml to wave 1. Both are listed as intentionally not onboarded in ci/render-golden.sh (RENDER_GOLDEN_NOT_ONBOARDED="keycloak vm-demo"); their manifests are tracked in apps/ but deliberately absent from the resources lists so Flux does not apply them. --emit-waves places any file present on disk and has no awareness of that exclusion, while render-golden.sh uses --wave-layout and compares rendered manifests only — it never checks kustomization resources, which is why this has survived unnoticed. So the header "DO NOT hand-edit: re-run render.py --emit-waves" is currently unsafe advice: following it onboards an IdP and a KubeVirt VM. I therefore restored both kustomizations and hand-added only my one line. I did not fix --emit-waves, and I have not determined how long the drift has existed or whether either service is meant to be onboarded.