Merge pull request 'feat(plex): one advertised hostname for local+remote, its own LE cert, and a LAN definition so LAN clients stop reading as remote' (#48) from feat/plex-vip-hostname-and-lan-networks into main
Reviewed-on: #48 Reviewed-by: Keith Jackson <[email protected]>
This commit was merged in pull request #48.
This commit is contained in:
jakman85 committed
commit
c163deaef9
11 files changed
+479
-85
No files matched your search
@@ -0,0 +1,151 @@
|
||||
---
|
||||
date: 2026-10-04
|
||||
branch: feat/plex-vip-hostname-and-lan-networks
|
||||
title: Plex — one advertised hostname for local and remote, its own LE certificate, and a LAN definition so LAN clients stop being reported as remote
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Two operator-reported symptoms, after the 2026-10-04 "server missing from the account" fix
|
||||
(decisions/2026-10-04-plex-advertise-plexdirect.md) had restored publishing:
|
||||
|
||||
1. **LAN clients were shown as Remote.** Root cause, measured: `LanNetworksBandwidth` was **ABSENT**,
|
||||
so Plex had no definition of a local network and classified *everything* as WAN — **468 requests
|
||||
logged `Allowed Network (WAN)` and ZERO `(LAN)`**, including Seerr dialling from inside the
|
||||
cluster. `allowedNetworks` (the `ALLOWED_NETWORKS` env) only waives **auth**; it does not make a
|
||||
client local. Two different settings, and only one of them has an env hook.
|
||||
2. **The advertised URL was a single-purpose, fragile value.** The previous fix advertised
|
||||
`https://<dashed-ip>.<serverhash>.plex.direct:32400`, which works, but hard-codes the VIP *and*
|
||||
the server's certificate hash into a cluster-var — two values nothing could keep in sync.
|
||||
|
||||
**Operator design adopted (2026-10-04): ONE hostname, `plex-vip.${BASE_DOMAIN}`, serving both
|
||||
audiences by split-horizon DNS** — PiHole answers it with `${PLEX_L4_VIP}` on the LAN, Cloudflare
|
||||
answers it publicly with the WAN address, NATed to the same VIP.
|
||||
|
||||
- **`catalog/plex.yaml`:** `ADVERTISE_IP` → `https://plex-vip.${BASE_DOMAIN}:32400`. New
|
||||
`lifecycle.postStart` hook (level-triggered, same idiom as pihole's and qbittorrent's) that
|
||||
re-asserts on **every** start: `LanNetworksBandwidth`, and the three `customCertificate*` prefs
|
||||
after converting the PEM to the PKCS#12 bundle Plex requires. New `PLEX_TLS_CRT`/`PLEX_TLS_KEY`
|
||||
env from the `plex-vip-tls` Secret, both `optional: true`.
|
||||
- **`apps/wave-0/plex-cert.yaml` (new, hand-written):** a cert-manager `Certificate` for exactly
|
||||
`plex-vip.${BASE_DOMAIN}` from ClusterIssuer `letsencrypt-prod`. Non-catalog because the schema has
|
||||
no field for a Certificate. A single name, not the wildcard: the wildcard's Secret lives in
|
||||
`gateway` and Secrets are not readable across namespaces, and re-requesting `*.${BASE_DOMAIN}`
|
||||
would burn a duplicate-certificate slot for no benefit.
|
||||
- **`schemas/servicedefinition.schema.json` → 1.11.0:** adds `optional` to
|
||||
`env.valueFrom.secretKeyRef` / `configMapKeyRef`. **This is load-bearing, not tidiness.** The
|
||||
Certificate must be issued into the `plex` namespace, and that namespace is created by the
|
||||
rendered `apps/wave-0/plex.yaml` itself — so it cannot be issued in an earlier tier and lands in
|
||||
the *same* reconcile as the workload. With hard refs, Plex would sit in
|
||||
`CreateContainerConfigError` until a certificate issued, and `apps-wave-0` is `wait: true` with
|
||||
`apps-wave-1` dependsOn it, so a slow or failed DNS01 challenge would have **stalled the whole app
|
||||
tier** — the hazard `catalog/plex.yaml` already documents for its GPU resource.
|
||||
- **`catalog/pihole.yaml`:** `address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}` added to
|
||||
`FTLCONF_misc_dnsmasq_lines`. Required, not cosmetic: measured 2026-10-04,
|
||||
`plex-vip.${BASE_DOMAIN}` resolved to **`192.168.2.20`** (the Gateway VIP) via the zone wildcard,
|
||||
and a connect to `:32400` there returned **`http=000`**.
|
||||
- **`clusters/base/cluster-vars.yaml` + `clusters/prod/kustomization.yaml`:** `PLEX_ADVERTISE_URL`
|
||||
**deleted** — the hostname makes it unnecessary, which removes the cert-hash fragility outright.
|
||||
`PLEX_L4_VIP`'s note now records its second consumer (pihole).
|
||||
- **`catalog/render.py`:** `plex-cert` EXTRA_UNITS entry; **`apps/wave-0/kustomization.yaml`:**
|
||||
`plex-cert.yaml` hand-added (NOT via `--emit-waves`, which would also onboard `keycloak` and
|
||||
`vm-demo` — the pre-existing drift recorded on 2026-10-03).
|
||||
|
||||
**A correction to my own earlier reasoning, recorded because it was wrong in the handoff log.** The
|
||||
previous entry asserted a bare-IP URL "can NEVER validate" and treated a SAN match as a
|
||||
*requirement*. The operator pointed out it is not: `httpsRequired=0` and `secureConnections` default
|
||||
to Preferred, so clients can fall back to cleartext. The measurements still stand (a validating
|
||||
client did fail, and real clients did send `certificate unknown`) — but the correct framing is that a
|
||||
matching certificate keeps the remote leg **encrypted**, not that it is required for Plex to
|
||||
function. That is why this branch obtains a real certificate for the advertised name rather than
|
||||
choosing the name to suit Plex's built-in one.
|
||||
|
||||
**SENSITIVE PATHS — touches `clusters/**` (two files), so `ci/approval-policy.sh` will BLOCK until an
|
||||
approving review (or the operator's `/approve-override <reason>` posted as a PR *review* of type
|
||||
"Comment"). Deliberately not self-posted: CLAUDE.md §1.**
|
||||
|
||||
## What I verified
|
||||
|
||||
Live prod, 2026-10-04, read-only. Secrets per CLAUDE.md §6: prefs were read by name-scoped grep,
|
||||
the server token was used inside the pod and never printed, and no `Preferences.xml` was dumped.
|
||||
|
||||
- **The LAN-classification root cause, directly:** `LanNetworksBandwidth=<ABSENT>`, and request
|
||||
tagging of **468 × `Allowed Network (WAN)` vs 0 × `(LAN)`** — including `10.0.2.33` (Seerr), an
|
||||
in-cluster pod. This is the measurement the whole hook rests on.
|
||||
- **All four prefs the hook sets EXIST on this build**, read from the server's own `/:/prefs`
|
||||
(token-authenticated, output filtered to ids): `LanNetworksBandwidth`, `customCertificatePath`,
|
||||
`customCertificateKey`, `customCertificateDomain`, all `type="text"`. I had earlier been unable to
|
||||
confirm this (an unauthenticated `GET /:/prefs` returns **401**) and said so rather than assuming.
|
||||
- **There is NO env path for these settings** — the reason the hook exists. Grep of the image's
|
||||
entire init surface: `setPref` is called for exactly six prefs (`MachineIdentifier`,
|
||||
`ProcessedMachineIdentifier`, `PlexOnlineToken`, `customConnections`, `allowedNetworks`,
|
||||
`TranscoderTempDirectory`) and the only `PLEX_*` env it reads are CLAIM/CLAIM_FILE/UID/GID/
|
||||
UPDATE_CHANNEL and five INFO/HOME vars. No `PLEX_PREFERENCE*` of any kind; zero files mention
|
||||
`LanNetworks`.
|
||||
- **The DNS gap was real:** `plex-vip.${BASE_DOMAIN}` resolved to `192.168.2.20` (Gateway VIP), and
|
||||
`https://192.168.2.20:32400` returned `http=000` (nothing listening). PiHole's wildcard is why.
|
||||
- **`openssl` is present in the image** (`/usr/bin/openssl`, 3.0.13) — the hook's PEM→PKCS#12
|
||||
conversion has a tool to run.
|
||||
- **The issuer exists and works:** ClusterIssuer `letsencrypt-prod` `READY=True` (28d), and the
|
||||
existing `gateway/wildcard-home` Certificate is `Ready`/not-expired, which is what makes a DNS01
|
||||
issuance for a new name in this zone a reasonable expectation.
|
||||
- **The renderer's substitution guard caught a real defect in my own work**, which is recorded
|
||||
because it is the second time this class of bug has appeared here: my hook's explanatory comment
|
||||
contained brace-wrapped names, and `assert_substitution_safe` refused to emit
|
||||
(`'PX' is not a declared cluster-vars key — envsubst will replace it with the EMPTY STRING`).
|
||||
Reworded; the rendered hook now contains exactly one brace-wrapped name, `BASE_DOMAIN`, verified
|
||||
by parsing the emitted manifest.
|
||||
- **A `set -u` bug I introduced and then removed.** With `optional: true` the TLS env vars are
|
||||
genuinely UNSET when no certificate has issued; under `set -u` the first `[ -n ... ]` would abort
|
||||
the hook, and a postStart hook exiting non-zero makes kubelet KILL the container — the exact
|
||||
failure `optional` exists to prevent. Verified by parsing the rendered manifest that no `set -u`
|
||||
**command** remains (only two mentions in comments).
|
||||
- **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with
|
||||
`spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all
|
||||
**pass**; the rc 3 is the two known environmental skips (`gitea-settings` — no local
|
||||
`SETTINGS_API_TOKEN`; `conformance` L2/L3 — `CLUSTER_AVAILABLE` unset). **Not green. Reported as
|
||||
rc 3.** `l0` passing is the specific proof the schema edit admits `optional` and still validates
|
||||
every catalog file; `render-golden` passing is the proof catalog and committed goldens agree.
|
||||
- **Only plex and pihole goldens moved.** No other service's manifest changed, and no service was
|
||||
silently onboarded.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **THE OUTCOME, ENTIRELY. Nothing in this branch has been observed working.** It is unreconciled:
|
||||
no Certificate exists in prod, `plex-vip` still resolves to the Gateway VIP, `LanNetworksBandwidth`
|
||||
is still absent, and the hook has NEVER RUN. Every claim above is about the inputs, not the result.
|
||||
- **Whether the certificate will actually ISSUE.** DNS01 for a new name in this zone is inferred from
|
||||
the wildcard's success, not demonstrated. If the Cloudflare credential or solver is scoped in a way
|
||||
that fails for this name, Plex will start (by design) with no TLS and the advertised `https://` URL
|
||||
will be unservable — i.e. **this branch could leave the advertised URL broken in a new way**. First
|
||||
reconcile must be watched.
|
||||
- **Whether `openssl pkcs12 -export` produces a bundle PLEX ACCEPTS.** OpenSSL 3 defaults to AES-256
|
||||
+ SHA-256, which some consumers reject. Untested against Plex. If it is rejected, Plex may fail to
|
||||
load its TLS listener — the hook guards against a *missing* file, NOT against a present-but-
|
||||
unreadable one. **This is the single largest unverified risk in the change.**
|
||||
- **Whether the certificate prefs take effect without a restart.** Plex may need to re-initialise its
|
||||
listener after `customCertificatePath` changes. If so, the first reconcile applies the prefs and
|
||||
the SECOND start serves the cert. Converges either way, but the first start may look wrong.
|
||||
- **Whether `LanNetworksBandwidth` actually flips the Remote badge.** It is the correct setting by
|
||||
Plex's own documentation and the measured absence explains the symptom exactly, but I did not set it
|
||||
live and observe a client reclassify.
|
||||
- **SNAT makes the LAN definition coarser than it looks.** With `externalTrafficPolicy: Cluster`, Plex
|
||||
sees the ingress NODE's address for everything arriving via the VIP, so once `10.0.0.0/8` and
|
||||
`192.168.0.0/16` are LAN, GENUINE REMOTE clients arriving that way will ALSO be classified local
|
||||
and remote bandwidth limits will not bind on them. This is the metering consequence already
|
||||
accepted on 2026-10-03, now made stronger rather than weaker. Not measured.
|
||||
- **The public Cloudflare record for `plex-vip` is off-repo and absent from this branch**, as is the
|
||||
WAN tcp/32400 port-forward. Remote access cannot work until the operator creates both; nothing here
|
||||
can verify either, and this repo has no off-LAN vantage point.
|
||||
- **The private key is exposed in the container's environment** (`/proc/1/environ`) because the schema
|
||||
and renderer support no secret VOLUME — measured: `storage` accepts only
|
||||
[class, export, mountPath, name, readOnly, size, subPath] with `additionalProperties: false`, and
|
||||
no rendered app mounts a Secret. A volume would be the better shape; adding secret-volume support
|
||||
is the clean follow-up and I did not do it.
|
||||
- **The PKCS#12 passphrase is a hardcoded literal** (`plexvip`) in the rendered manifest. It protects
|
||||
nothing — Plex needs it in cleartext in `customCertificateKey` and it sits beside the bundle on the
|
||||
same PVC — but it is worth knowing it is not a secret and was never treated as one.
|
||||
- **Four places must agree and nothing enforces it:** `ADVERTISE_IP`, the hook's
|
||||
`customCertificateDomain`, `plex-cert.yaml`'s `dnsNames`, and PiHole's answer. No gate stage or
|
||||
conformance test checks that they match, or that PiHole's `plex-vip` answer still equals
|
||||
`${PLEX_L4_VIP}`.
|
||||
Reference in new issue
Block a user