feat(plex): expose tcp/32400 for remote access via a pinned LoadBalancer VIP #43

2026-10-04T04:33:33Z

gitea-admin commented

2026-10-04 04:33:33 +00:00

Reported symptom: users cannot reach Plex from outside the LAN. Two independent causes, both measured against live prod on 2026-10-03.

What was actually wrong

Plex advertised no remote connection at all. Live Preferences.xml: customConnections ABSENT, ManualPortMappingMode ABSENT, while PublishServerOnPlexOnlineKey="1" — publishing was on, with nothing reachable to publish. Plex's automatic UPnP/NAT-PMP path cannot work from a pod on the Cilium overlay.
Nothing from the WAN reached the cluster. 136.51.115.252:32400 → refused. 136.51.115.252:443 → a self-signed router cert (C=ZZ, CN=%D), not the Gateway's LE wildcard. The Plex Service was ClusterIP only, so 32400 had no off-cluster path at all.

The L7 Gateway was NOT the problem. It serves Plex correctly — GET /identity → HTTP 200, claimed="1", and 401 on every protected endpoint. But Plex apps discover servers through plex.tv, not DNS, so an HTTPRoute can never put this server on a client's connection list.

What this changes

A hand-written non-catalog unit, apps/wave-0/plex-remote.yaml: a type=LoadBalancer Service on tcp/32400 pinned to ${PLEX_L4_VIP} plus a CiliumNetworkPolicy admitting world/cluster/host/remote-node. Non-catalog because the SD schema is additionalProperties: false and cannot express a LoadBalancer Service or a pinned lbipam VIP — the same reason pihole-dns.yaml, crafty/service-l4.yaml and unifi/50-service-l4.yaml are hand-written. Registered in render.py's EXTRA_UNITS so --emit-waves places it in plex's own wave.

PLEX_L4_VIP = 192.168.2.24 on prod, verified free: pool default-pool IPsTotal 11 / IPsUsed 4 / IPsAvailable 7, allocations only .20/.21/.22/.23.

ADVERTISE_IP deliberately not set. Advertising the Gateway URL was the alternative and the mechanism was verified to work on an already-claimed server. Operator chose the native path so Plex terminates its own TLS (*.plex.direct) and streams bypass the Envoy L7 hop.

externalTrafficPolicy: Cluster, inheriting crafty's 8-day-outage lesson rather than re-deriving it. Documented consequence: SNAT puts remote clients inside ALLOWED_NETWORKS, so Plex meters them as LAN (confirmed — a probe logged as (Allowed Network (WAN))). Metering effect, not an authorization change.

⚠️ This does not work yet — two manual steps remain

Neither can live in this repo. runbooks/plex-remote-access.md is the procedure.

Router: forward WAN tcp/32400 → 192.168.2.24. Also check whether the router's management UI is bound to WAN:443, which would shadow the Gateway's forward.
Plex UI: Settings → Remote Access → tick "manually specify public port" = 32400. There is no env var for this — grep of the running image's entire init surface for ManualPortMapping/PLEX_PREFERENCE returned NONE.

Gate

./infra/devcontainer/dev.sh run bash -c './ci/gate.sh' → rc 3, not green, reported as rc 3.

spec-immutability, decisions-log, lint, l0, render-golden, l1, approval-policy all pass. The rc 3 comes from two environmental skips only: gitea-settings (no local SETTINGS_API_TOKEN) and conformance L2/L3 (CLUSTER_AVAILABLE unset). render-golden passing is the specific proof that the render.py change, the new unit and the edited wave kustomization are mutually consistent.

🔒 Sensitive path — needs an operator override

Touches clusters/** (2 files), so ci/approval-policy.sh will block. Per CLAUDE.md §1 I have not posted the override myself — the agent account holds owner, so that prohibition is a rule rather than a limit. Please post, as a PR review of type "Comment":

/approve-override <reason>

Not verified — please read

The fix itself is unproven. Both remaining steps are off-repo and manual. This removes two blockers; it does not demonstrate an outcome.
Outside-in reachability, at all. Every probe was from inside the LAN, so the :443 router cert strictly proves only that there is no NAT hairpin — not outside-in failure. There is no off-LAN vantage point here.
192.168.2.24 was never ARP-probed off-cluster (only in-cluster pool accounting). .23 was arping-probed before it was claimed; .24 was not.
The LoadBalancer has never been reconciled — LB-IPAM has not assigned the VIP, L2 announcement is unobserved, and the new policy has never admitted a real packet. The entity set is reasoned from crafty's working game-port rule, not measured.

Pre-existing drift found and deliberately left alone

Running the documented render.py --emit-waves wants to add vm-demo.yaml to wave 0 and keycloak.yaml to wave 1 — both listed as intentionally not onboarded in ci/render-golden.sh. --emit-waves places any file on disk with no awareness of that exclusion, and render-golden.sh uses --wave-layout and never checks kustomization resources, which is why it has survived unnoticed. So the "DO NOT hand-edit: re-run --emit-waves" header is currently unsafe advice — following it onboards an IdP and a KubeVirt VM. I restored both kustomizations and hand-added only my one line. Not fixed here; worth its own issue.

🤖 Generated with Claude Code

Reported symptom: **users cannot reach Plex from outside the LAN.** Two independent causes, both measured against live prod on 2026-10-03. ### What was actually wrong 1. **Plex advertised no remote connection at all.** Live `Preferences.xml`: `customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while `PublishServerOnPlexOnlineKey="1"` — publishing was on, with nothing reachable to publish. Plex's automatic UPnP/NAT-PMP path cannot work from a pod on the Cilium overlay. 2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` → refused. `136.51.115.252:443` → a **self-signed router cert** (`C=ZZ, CN=%D`), not the Gateway's LE wildcard. The Plex Service was `ClusterIP` only, so 32400 had no off-cluster path at all. **The L7 Gateway was NOT the problem.** It serves Plex correctly — `GET /identity` → `HTTP 200`, `claimed="1"`, and `401` on every protected endpoint. But Plex apps discover servers through **plex.tv**, not DNS, so an HTTPRoute can never put this server on a client's connection list. ### What this changes A hand-written non-catalog unit, `apps/wave-0/plex-remote.yaml`: a `type=LoadBalancer` Service on tcp/32400 pinned to `${PLEX_L4_VIP}` plus a `CiliumNetworkPolicy` admitting `world`/`cluster`/`host`/`remote-node`. Non-catalog because the SD schema is `additionalProperties: false` and cannot express a LoadBalancer Service or a pinned lbipam VIP — the same reason `pihole-dns.yaml`, `crafty/service-l4.yaml` and `unifi/50-service-l4.yaml` are hand-written. Registered in `render.py`'s `EXTRA_UNITS` so `--emit-waves` places it in plex's own wave. `PLEX_L4_VIP` = `192.168.2.24` on prod, verified free: pool `default-pool` IPsTotal 11 / IPsUsed 4 / IPsAvailable 7, allocations only `.20`/`.21`/`.22`/`.23`. **`ADVERTISE_IP` deliberately not set.** Advertising the Gateway URL was the alternative and the mechanism was verified to work on an already-claimed server. Operator chose the native path so Plex terminates its own TLS (`*.plex.direct`) and streams bypass the Envoy L7 hop. **`externalTrafficPolicy: Cluster`**, inheriting crafty's 8-day-outage lesson rather than re-deriving it. Documented consequence: SNAT puts remote clients inside `ALLOWED_NETWORKS`, so **Plex meters them as LAN** (confirmed — a probe logged as `(Allowed Network (WAN))`). Metering effect, not an authorization change. ### ⚠️ This does not work yet — two manual steps remain Neither can live in this repo. `runbooks/plex-remote-access.md` is the procedure. 1. **Router:** forward WAN tcp/32400 → `192.168.2.24`. Also check whether the router's management UI is bound to WAN:443, which would shadow the Gateway's forward. 2. **Plex UI:** Settings → Remote Access → tick **"manually specify public port"** = 32400. There is **no env var for this** — grep of the running image's entire init surface for `ManualPortMapping`/`PLEX_PREFERENCE` returned **NONE**. ### Gate `./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'` → **rc 3**, not green, reported as rc 3. `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1`, `approval-policy` all **pass**. The rc 3 comes from two environmental skips only: `gitea-settings` (no local `SETTINGS_API_TOKEN`) and `conformance` L2/L3 (`CLUSTER_AVAILABLE` unset). `render-golden` passing is the specific proof that the `render.py` change, the new unit and the edited wave kustomization are mutually consistent. ### 🔒 Sensitive path — needs an operator override Touches `clusters/**` (2 files), so `ci/approval-policy.sh` will **block**. Per CLAUDE.md §1 I have **not** posted the override myself — the agent account holds `owner`, so that prohibition is a rule rather than a limit. Please post, as a PR **review** of type "Comment": ``` /approve-override <reason> ``` ### Not verified — please read - **The fix itself is unproven.** Both remaining steps are off-repo and manual. This removes two blockers; it does not demonstrate an outcome. - **Outside-in reachability, at all.** Every probe was from inside the LAN, so the `:443` router cert strictly proves only that there is **no NAT hairpin** — not outside-in failure. There is no off-LAN vantage point here. - **`192.168.2.24` was never ARP-probed off-cluster** (only in-cluster pool accounting). `.23` was `arping`-probed before it was claimed; `.24` was not. - **The LoadBalancer has never been reconciled** — LB-IPAM has not assigned the VIP, L2 announcement is unobserved, and the new policy has never admitted a real packet. The entity set is reasoned from crafty's working game-port rule, not measured. ### Pre-existing drift found and deliberately left alone Running the documented `render.py --emit-waves` wants to add **`vm-demo.yaml`** to wave 0 and **`keycloak.yaml`** to wave 1 — both listed as intentionally *not* onboarded in `ci/render-golden.sh`. `--emit-waves` places any file on disk with no awareness of that exclusion, and `render-golden.sh` uses `--wave-layout` and never checks kustomization `resources`, which is why it has survived unnoticed. **So the "DO NOT hand-edit: re-run `--emit-waves`" header is currently unsafe advice** — following it onboards an IdP and a KubeVirt VM. I restored both kustomizations and hand-added only my one line. Not fixed here; worth its own issue. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea-admin added 1 commit 2026-10-04 04:33:33 +00:00

feat(plex): expose tcp/32400 for remote access via a pinned LoadBalancer VIP

gate / gate (pull_request) Successful in 1m49s

Details

gate / conformance (pull_request) Has been skipped

Details

fbaeb3e038

Remote Plex clients had no path in, for two measured reasons: Plex advertised
no remote connection at all (customConnections and ManualPortMappingMode both
ABSENT while PublishServerOnPlexOnlineKey=1), and its Service was ClusterIP, so
32400 had no off-cluster path. The L7 Gateway was NOT the problem -- it serves
Plex correctly (200, claimed=1, 401 on protected endpoints) -- but Plex clients
discover servers through plex.tv, not DNS, so an HTTPRoute cannot put this
server on a client's connection list.

Operator chose the native path over ADVERTISE_IP: Plex terminates its own TLS
with *.plex.direct and streams bypass the Envoy L7 hop.

Adds a hand-written non-catalog unit (the SD schema cannot express a
LoadBalancer Service or a pinned lbipam VIP), registered in render.py's
EXTRA_UNITS so --emit-waves places it in plex's own wave.

externalTrafficPolicy: Cluster, inheriting crafty's 8-day-outage lesson rather
than re-deriving it. Known consequence, documented: SNAT puts remote clients
inside ALLOWED_NETWORKS, so Plex meters them as LAN. Not an auth change.

Remote access does NOT work until the operator does two manual steps that
cannot live in this repo -- the WAN tcp/32400 forward and Plex's "manually
specify public port" toggle (no env var exists for it; verified NONE in the
image's init surface). runbooks/plex-remote-access.md is the procedure.

Touches clusters/** -- a sensitive path -- so the approval-policy stage will
block pending an operator /approve-override.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

jakman85 approved these changes

2026-10-04 04:42:31 +00:00

jakman85 merged commit 939461120a into main

2026-10-04 04:46:34 +00:00

jakman85 referenced this issue from a commit

2026-10-04 04:46:36 +00:00

Merge pull request 'feat(plex): expose tcp/32400 for remote access via a pinned LoadBalancer VIP' (#43) from feat/plex-remote-access-l4 into main

gitea-admin referenced this pull request

2026-10-04 06:35:34 +00:00

feat(pihole): two replicas on different nodes, and fix a PVC mounted where pihole never writes #45

gitea-admin referenced this pull request

2026-10-04 07:04:09 +00:00