--- date: 2026-10-03 branch: feat/plex-remote-access-l4 title: Plex remote access — raw tcp/32400 via a pinned LoadBalancer VIP, not the L7 Gateway --- ## What changed Reported symptom: users cannot reach Plex from outside the LAN. **Two independent causes**, measured rather than inferred: 1. **Plex advertised no remote connection at all.** Live `Preferences.xml` on prod: `customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while `PublishServerOnPlexOnlineKey="1"` — publishing was on and there was nothing reachable to publish. Plex's automatic UPnP/NAT-PMP path cannot work from a pod on the Cilium overlay. 2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` refused; `136.51.115.252:443` answered with a **self-signed router cert** (`C=ZZ, CN=%D`), not the Gateway's LE wildcard. The Plex Service was `ClusterIP` only — no off-cluster path to 32400 by any route. This branch fixes the in-cluster half. **Remote access does not work until the operator also does two manual steps** (`runbooks/plex-remote-access.md`): the WAN tcp/32400 port-forward, and ticking "manually specify public port" in the Plex UI. - **`apps/wave-0/plex-remote.yaml` (new, hand-written).** A `type=LoadBalancer` Service on tcp/32400 pinned to `${PLEX_L4_VIP}` via `lbipam.cilium.io/ips`, plus a `CiliumNetworkPolicy` (`plex-allow-remote-32400`) admitting `world`/`cluster`/`host`/`remote-node` to 32400. Non-catalog because `schemas/servicedefinition.schema.json` is `additionalProperties: false` and has no field for a LoadBalancer Service or a pinned VIP — the same reason `apps/wave-0/pihole-dns.yaml`, `platform/crafty/service-l4.yaml` and `apps/vms/unifi/50-service-l4.yaml` are hand-written. - **`catalog/render.py`:** an `EXTRA_UNITS` entry (`plex-remote`, anchored to `plex`, `above: False`) so `--emit-waves` places the unit in plex's own wave. - **`clusters/base/cluster-vars.yaml`:** new `PLEX_L4_VIP` (dev default `192.168.1.57`), plus the substitution inventory comment updated to name `plex-remote: PLEX_L4_VIP`. - **`clusters/prod/kustomization.yaml`:** prod override `PLEX_L4_VIP: 192.168.2.24`. - **`apps/wave-0/kustomization.yaml`:** `plex-remote.yaml` added to `resources`. - **`runbooks/plex-remote-access.md` (new):** the two manual steps, the four-step verification, why no alert is possible, and the two documented warts below. **`ADVERTISE_IP` was deliberately NOT set.** Setting it to `https://plex.home.keithjackson.dev:443` was the alternative — it needs no new WAN hole and the mechanism was verified working on an already-claimed server. **The operator chose the native path (2026-10-03)** so Plex terminates its own TLS (`*.plex.direct`) and streams bypass the Envoy L7 hop. `catalog/plex.yaml`'s deferral note for `ADVERTISE_IP` remains accurate and untouched. **`externalTrafficPolicy: Cluster`, with a known cost.** `Local` would preserve the client IP, but Cilium's L2 announcer does not follow the backend pod — the wart that cost this LAN an 8-day Minecraft outage (see `platform/crafty/service-l4.yaml`). Consequence accepted and documented: SNAT puts the client in `192.168.0.0/16`, inside `ALLOWED_NETWORKS`, so **Plex classifies remote streams as LAN**. That is a metering effect (remote bandwidth limits will not bind), not an authorization one. **SENSITIVE PATHS — this PR touches `clusters/**` (two files), so `ci/approval-policy.sh` will BLOCK it.** It needs the operator's `/approve-override ` posted as a PR **review** of type "Comment". I have deliberately not posted it: CLAUDE.md §1 forbids an agent clearing this on its own PR, and the agent account holds `owner`, so the prohibition is a rule, not a limit. ## What I verified All measurements against live prod, 2026-10-03, read-only. - **Plex's advertised state**, the finding this branch rests on: `customConnections=`, `ManualPortMappingMode=`, `ManualPortMappingPort=`, `PublishServerOnPlexOnlineKey="1"`, `allowedNetworks="10.0.0.0/8,192.168.0.0/16"`. Read by attribute grep, never by dumping `Preferences.xml` (it holds `PlexOnlineToken`). - **The L7 Gateway path works and is NOT the problem.** Via the Gateway VIP, `GET https://plex.home.keithjackson.dev/identity` → `HTTP 200`, `claimed="1"`, real `machineIdentifier`. HTTPRoute `plex-32400`: `Accepted=True`, `ResolvedRefs=True`. - **Plex enforces its own auth**, which is what makes an unfiltered WAN port defensible: `/library/sections`, `/status/sessions`, `/:/prefs` all → **401** unauthenticated through the Gateway. - **The SNAT/allowedNetworks wart is real, not theoretical.** A tagged probe through the Gateway was logged by Plex as `[10.0.2.214:36040 (Allowed Network (WAN))]` — the Envoy pod IP, classified as an allowed network. - **WAN state:** `136.51.115.252:32400` → connection refused. `136.51.115.252:443` → self-signed cert `C=ZZ, ST=Somewhere, O=%D342e7e6b, CN=%D`. Public DNS resolves correctly: `plex.home.keithjackson.dev` → CNAME `wan.home.keithjackson.dev` → `136.51.115.252`. - **`192.168.2.24` is free in-cluster.** `CiliumLoadBalancerIPPool default-pool`: IPsTotal 11 / IPsUsed 4 / IPsAvailable 7; the only allocations are `.20` (cilium-gateway-shared), `.21` (unifi-l4-*), `.22` (crafty-l4-*), `.23` (pihole-dns-*). Not a node address (`.10/.11/.12`) and not the kube-vip control-plane VIP (`.9`). - **The manual-port-mapping step genuinely cannot be automated.** Grep of the running image's whole init surface (`/etc/cont-init.d`, `/plex-common.sh`, `/usr/local/bin`) for `ManualPortMapping` and `PLEX_PREFERENCE` → **NONE**. - **`ADVERTISE_IP` would have worked on a claimed server** (recorded because the runbook's "why not" rests on it): `setPref "customConnections"` at line 126 of `40-plex-first-run` sits **outside** the `PLEX_CLAIM`/`PlexOnlineToken` guard, and the `/.firstRunComplete` early-exit marker is on the **overlay** FS, not the `/config` PVC, so the script re-runs every pod start. - **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**. The rc 3 is two environmental skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`) and `conformance` L2/L3 (`CLUSTER_AVAILABLE` unset). **Not green. Reported as rc 3, not as pass.** - **`render-golden` passing is the specific proof** that the `render.py` `EXTRA_UNITS` change, the new hand-written unit and the edited wave kustomization are mutually consistent. ## What I did NOT verify - **THE ACTUAL FIX. Nothing in this branch has been proven to make remote access work** — it cannot be, because both remaining steps are off-repo and manual (router forward, Plex UI toggle). This branch removes two blockers and adds none; it does not demonstrate an outcome. - **Outside-in reachability, at all.** Every probe above was taken from inside the LAN, so the `:443` self-signed cert strictly proves only that there is **no NAT hairpin** — it does **not** prove outside-in failure. A router cert on WAN:443 is suggestive (its management UI would shadow a forward), but unproven. **This repo has no off-LAN vantage point.** Test 3 in the runbook (app.plex.tv from a cellular connection) is the only honest check. - **`192.168.2.24` was never ARP-probed off-cluster.** Only in-cluster pool accounting was checked. `.23` was `arping`-probed before it was claimed; `.24` was not, so an off-cluster squatter would not have been caught. - **The LoadBalancer has never been reconciled.** `plex-l4-tcp` does not exist in prod yet, so LB-IPAM has not actually assigned the pinned VIP, L2 announcement has not been observed for it, and the new CiliumNetworkPolicy has never admitted a real packet. Whether `world`/`host`/`cluster`/ `remote-node` is the correct entity set **under this SNAT path** is reasoned from `platform/crafty/networkpolicy.yaml`'s working game-port rule, **not measured**. - **Whether Plex's remote path needs anything beyond the manual port tick.** If plex.tv still lists no remote connection after both manual steps, this branch is necessary-but-insufficient. - **The CI approval-policy outcome.** The local stage reported green, but that is an offline N/A, not a verdict: this PR touches `clusters/**`, so CI will block it pending an override. - **PRE-EXISTING DRIFT I FOUND AND DELIBERATELY LEFT ALONE — worth its own follow-up.** Running the documented `python3 catalog/render.py --emit-waves` regenerates the wave kustomizations and wants to add **`vm-demo.yaml`** to wave 0 and **`keycloak.yaml`** to wave 1. Both are listed as intentionally not onboarded in `ci/render-golden.sh` (`RENDER_GOLDEN_NOT_ONBOARDED="keycloak vm-demo"`); their manifests are tracked in `apps/` but deliberately absent from the `resources` lists so Flux does not apply them. `--emit-waves` places any file present on disk and has **no awareness of that exclusion**, while `render-golden.sh` uses `--wave-layout` and compares rendered manifests only — it never checks kustomization `resources`, which is why this has survived unnoticed. **So the header "DO NOT hand-edit: re-run `render.py --emit-waves`" is currently unsafe advice**: following it onboards an IdP and a KubeVirt VM. I therefore restored both kustomizations and hand-added only my one line. I did not fix `--emit-waves`, and I have not determined how long the drift has existed or whether either service is *meant* to be onboarded.