home-network-k3s-impl/decisions/2026-10-03-plex-remote-access-l4.md

---
date: 2026-10-03
branch: feat/plex-remote-access-l4
title: Plex remote access — raw tcp/32400 via a pinned LoadBalancer VIP, not the L7 Gateway
---

## What changed

Reported symptom: users cannot reach Plex from outside the LAN. **Two independent causes**, measured
rather than inferred:

1. **Plex advertised no remote connection at all.** Live `Preferences.xml` on prod:
   `customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while
   `PublishServerOnPlexOnlineKey="1"` — publishing was on and there was nothing reachable to publish.
   Plex's automatic UPnP/NAT-PMP path cannot work from a pod on the Cilium overlay.
2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` refused;
   `136.51.115.252:443` answered with a **self-signed router cert** (`C=ZZ, CN=%D`), not the
   Gateway's LE wildcard. The Plex Service was `ClusterIP` only — no off-cluster path to 32400 by
   any route.

This branch fixes the in-cluster half. **Remote access does not work until the operator also does
two manual steps** (`runbooks/plex-remote-access.md`): the WAN tcp/32400 port-forward, and ticking
"manually specify public port" in the Plex UI.

- **`apps/wave-0/plex-remote.yaml` (new, hand-written).** A `type=LoadBalancer` Service on tcp/32400
  pinned to `${PLEX_L4_VIP}` via `lbipam.cilium.io/ips`, plus a `CiliumNetworkPolicy`
  (`plex-allow-remote-32400`) admitting `world`/`cluster`/`host`/`remote-node` to 32400. Non-catalog
  because `schemas/servicedefinition.schema.json` is `additionalProperties: false` and has no field
  for a LoadBalancer Service or a pinned VIP — the same reason `apps/wave-0/pihole-dns.yaml`,
  `platform/crafty/service-l4.yaml` and `apps/vms/unifi/50-service-l4.yaml` are hand-written.
- **`catalog/render.py`:** an `EXTRA_UNITS` entry (`plex-remote`, anchored to `plex`, `above: False`)
  so `--emit-waves` places the unit in plex's own wave.
- **`clusters/base/cluster-vars.yaml`:** new `PLEX_L4_VIP` (dev default `192.168.1.57`), plus the
  substitution inventory comment updated to name `plex-remote: PLEX_L4_VIP`.
- **`clusters/prod/kustomization.yaml`:** prod override `PLEX_L4_VIP: 192.168.2.24`.
- **`apps/wave-0/kustomization.yaml`:** `plex-remote.yaml` added to `resources`.
- **`runbooks/plex-remote-access.md` (new):** the two manual steps, the four-step verification, why
  no alert is possible, and the two documented warts below.

**`ADVERTISE_IP` was deliberately NOT set.** Setting it to `https://plex.home.keithjackson.dev:443`
was the alternative — it needs no new WAN hole and the mechanism was verified working on an
already-claimed server. **The operator chose the native path (2026-10-03)** so Plex terminates its own
TLS (`*.plex.direct`) and streams bypass the Envoy L7 hop. `catalog/plex.yaml`'s deferral note for
`ADVERTISE_IP` remains accurate and untouched.

**`externalTrafficPolicy: Cluster`, with a known cost.** `Local` would preserve the client IP, but
Cilium's L2 announcer does not follow the backend pod — the wart that cost this LAN an 8-day
Minecraft outage (see `platform/crafty/service-l4.yaml`). Consequence accepted and documented: SNAT
puts the client in `192.168.0.0/16`, inside `ALLOWED_NETWORKS`, so **Plex classifies remote streams
as LAN**. That is a metering effect (remote bandwidth limits will not bind), not an authorization one.

**SENSITIVE PATHS — this PR touches `clusters/**` (two files), so `ci/approval-policy.sh` will BLOCK
it.** It needs the operator's `/approve-override <reason>` posted as a PR **review** of type
"Comment". I have deliberately not posted it: CLAUDE.md §1 forbids an agent clearing this on its own
PR, and the agent account holds `owner`, so the prohibition is a rule, not a limit.

## What I verified

All measurements against live prod, 2026-10-03, read-only.

- **Plex's advertised state**, the finding this branch rests on: `customConnections=<ABSENT>`,
  `ManualPortMappingMode=<ABSENT>`, `ManualPortMappingPort=<ABSENT>`,
  `PublishServerOnPlexOnlineKey="1"`, `allowedNetworks="10.0.0.0/8,192.168.0.0/16"`. Read by
  attribute grep, never by dumping `Preferences.xml` (it holds `PlexOnlineToken`).
- **The L7 Gateway path works and is NOT the problem.** Via the Gateway VIP,
  `GET https://plex.home.keithjackson.dev/identity` → `HTTP 200`, `claimed="1"`, real
  `machineIdentifier`. HTTPRoute `plex-32400`: `Accepted=True`, `ResolvedRefs=True`.
- **Plex enforces its own auth**, which is what makes an unfiltered WAN port defensible:
  `/library/sections`, `/status/sessions`, `/:/prefs` all → **401** unauthenticated through the
  Gateway.
- **The SNAT/allowedNetworks wart is real, not theoretical.** A tagged probe through the Gateway was
  logged by Plex as `[10.0.2.214:36040 (Allowed Network (WAN))]` — the Envoy pod IP, classified as an
  allowed network.
- **WAN state:** `136.51.115.252:32400` → connection refused. `136.51.115.252:443` → self-signed
  cert `C=ZZ, ST=Somewhere, O=%D342e7e6b, CN=%D`. Public DNS resolves correctly:
  `plex.home.keithjackson.dev` → CNAME `wan.home.keithjackson.dev` → `136.51.115.252`.
- **`192.168.2.24` is free in-cluster.** `CiliumLoadBalancerIPPool default-pool`: IPsTotal 11 /
  IPsUsed 4 / IPsAvailable 7; the only allocations are `.20` (cilium-gateway-shared), `.21`
  (unifi-l4-*), `.22` (crafty-l4-*), `.23` (pihole-dns-*). Not a node address (`.10/.11/.12`) and not
  the kube-vip control-plane VIP (`.9`).
- **The manual-port-mapping step genuinely cannot be automated.** Grep of the running image's whole
  init surface (`/etc/cont-init.d`, `/plex-common.sh`, `/usr/local/bin`) for `ManualPortMapping` and
  `PLEX_PREFERENCE` → **NONE**.
- **`ADVERTISE_IP` would have worked on a claimed server** (recorded because the runbook's
  "why not" rests on it): `setPref "customConnections"` at line 126 of `40-plex-first-run` sits
  **outside** the `PLEX_CLAIM`/`PlexOnlineToken` guard, and the `/.firstRunComplete` early-exit
  marker is on the **overlay** FS, not the `/config` PVC, so the script re-runs every pod start.
- **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with
  `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all
  **pass**. The rc 3 is two environmental skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`)
  and `conformance` L2/L3 (`CLUSTER_AVAILABLE` unset). **Not green. Reported as rc 3, not as pass.**
- **`render-golden` passing is the specific proof** that the `render.py` `EXTRA_UNITS` change, the new
  hand-written unit and the edited wave kustomization are mutually consistent.

## What I did NOT verify

- **THE ACTUAL FIX. Nothing in this branch has been proven to make remote access work** — it cannot
  be, because both remaining steps are off-repo and manual (router forward, Plex UI toggle). This
  branch removes two blockers and adds none; it does not demonstrate an outcome.
- **Outside-in reachability, at all.** Every probe above was taken from inside the LAN, so the
  `:443` self-signed cert strictly proves only that there is **no NAT hairpin** — it does **not**
  prove outside-in failure. A router cert on WAN:443 is suggestive (its management UI would shadow a
  forward), but unproven. **This repo has no off-LAN vantage point.** Test 3 in the runbook
  (app.plex.tv from a cellular connection) is the only honest check.
- **`192.168.2.24` was never ARP-probed off-cluster.** Only in-cluster pool accounting was checked.
  `.23` was `arping`-probed before it was claimed; `.24` was not, so an off-cluster squatter would
  not have been caught.
- **The LoadBalancer has never been reconciled.** `plex-l4-tcp` does not exist in prod yet, so
  LB-IPAM has not actually assigned the pinned VIP, L2 announcement has not been observed for it, and
  the new CiliumNetworkPolicy has never admitted a real packet. Whether `world`/`host`/`cluster`/
  `remote-node` is the correct entity set **under this SNAT path** is reasoned from
  `platform/crafty/networkpolicy.yaml`'s working game-port rule, **not measured**.
- **Whether Plex's remote path needs anything beyond the manual port tick.** If plex.tv still lists
  no remote connection after both manual steps, this branch is necessary-but-insufficient.
- **The CI approval-policy outcome.** The local stage reported green, but that is an offline N/A, not
  a verdict: this PR touches `clusters/**`, so CI will block it pending an override.
- **PRE-EXISTING DRIFT I FOUND AND DELIBERATELY LEFT ALONE — worth its own follow-up.** Running the
  documented `python3 catalog/render.py --emit-waves` regenerates the wave kustomizations and wants to
  add **`vm-demo.yaml`** to wave 0 and **`keycloak.yaml`** to wave 1. Both are listed as
  intentionally not onboarded in `ci/render-golden.sh`
  (`RENDER_GOLDEN_NOT_ONBOARDED="keycloak vm-demo"`); their manifests are tracked in `apps/` but
  deliberately absent from the `resources` lists so Flux does not apply them. `--emit-waves` places
  any file present on disk and has **no awareness of that exclusion**, while `render-golden.sh` uses
  `--wave-layout` and compares rendered manifests only — it never checks kustomization `resources`,
  which is why this has survived unnoticed. **So the header "DO NOT hand-edit: re-run
  `render.py --emit-waves`" is currently unsafe advice**: following it onboards an IdP and a KubeVirt
  VM. I therefore restored both kustomizations and hand-added only my one line. I did not fix
  `--emit-waves`, and I have not determined how long the drift has existed or whether either service
  is *meant* to be onboarded.