home-network-k3s-impl/decisions/2026-10-04-plex-vip-hostname-and-lan-networks.md

---
date: 2026-10-04
branch: feat/plex-vip-hostname-and-lan-networks
title: Plex — one advertised hostname for local and remote, its own LE certificate, and a LAN definition so LAN clients stop being reported as remote
---

## What changed

Two operator-reported symptoms, after the 2026-10-04 "server missing from the account" fix
(decisions/2026-10-04-plex-advertise-plexdirect.md) had restored publishing:

1. **LAN clients were shown as Remote.** Root cause, measured: `LanNetworksBandwidth` was **ABSENT**,
   so Plex had no definition of a local network and classified *everything* as WAN — **468 requests
   logged `Allowed Network (WAN)` and ZERO `(LAN)`**, including Seerr dialling from inside the
   cluster. `allowedNetworks` (the `ALLOWED_NETWORKS` env) only waives **auth**; it does not make a
   client local. Two different settings, and only one of them has an env hook.
2. **The advertised URL was a single-purpose, fragile value.** The previous fix advertised
   `https://<dashed-ip>.<serverhash>.plex.direct:32400`, which works, but hard-codes the VIP *and*
   the server's certificate hash into a cluster-var — two values nothing could keep in sync.

**Operator design adopted (2026-10-04): ONE hostname, `plex-vip.${BASE_DOMAIN}`, serving both
audiences by split-horizon DNS** — PiHole answers it with `${PLEX_L4_VIP}` on the LAN, Cloudflare
answers it publicly with the WAN address, NATed to the same VIP.

- **`catalog/plex.yaml`:** `ADVERTISE_IP` → `https://plex-vip.${BASE_DOMAIN}:32400`. New
  `lifecycle.postStart` hook (level-triggered, same idiom as pihole's and qbittorrent's) that
  re-asserts on **every** start: `LanNetworksBandwidth`, and the three `customCertificate*` prefs
  after converting the PEM to the PKCS#12 bundle Plex requires. New `PLEX_TLS_CRT`/`PLEX_TLS_KEY`
  env from the `plex-vip-tls` Secret, both `optional: true`.
- **`apps/wave-0/plex-cert.yaml` (new, hand-written):** a cert-manager `Certificate` for exactly
  `plex-vip.${BASE_DOMAIN}` from ClusterIssuer `letsencrypt-prod`. Non-catalog because the schema has
  no field for a Certificate. A single name, not the wildcard: the wildcard's Secret lives in
  `gateway` and Secrets are not readable across namespaces, and re-requesting `*.${BASE_DOMAIN}`
  would burn a duplicate-certificate slot for no benefit.
- **`schemas/servicedefinition.schema.json` → 1.11.0:** adds `optional` to
  `env.valueFrom.secretKeyRef` / `configMapKeyRef`. **This is load-bearing, not tidiness.** The
  Certificate must be issued into the `plex` namespace, and that namespace is created by the
  rendered `apps/wave-0/plex.yaml` itself — so it cannot be issued in an earlier tier and lands in
  the *same* reconcile as the workload. With hard refs, Plex would sit in
  `CreateContainerConfigError` until a certificate issued, and `apps-wave-0` is `wait: true` with
  `apps-wave-1` dependsOn it, so a slow or failed DNS01 challenge would have **stalled the whole app
  tier** — the hazard `catalog/plex.yaml` already documents for its GPU resource.
- **`catalog/pihole.yaml`:** `address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}` added to
  `FTLCONF_misc_dnsmasq_lines`. Required, not cosmetic: measured 2026-10-04,
  `plex-vip.${BASE_DOMAIN}` resolved to **`192.168.2.20`** (the Gateway VIP) via the zone wildcard,
  and a connect to `:32400` there returned **`http=000`**.
- **`clusters/base/cluster-vars.yaml` + `clusters/prod/kustomization.yaml`:** `PLEX_ADVERTISE_URL`
  **deleted** — the hostname makes it unnecessary, which removes the cert-hash fragility outright.
  `PLEX_L4_VIP`'s note now records its second consumer (pihole).
- **`catalog/render.py`:** `plex-cert` EXTRA_UNITS entry; **`apps/wave-0/kustomization.yaml`:**
  `plex-cert.yaml` hand-added (NOT via `--emit-waves`, which would also onboard `keycloak` and
  `vm-demo` — the pre-existing drift recorded on 2026-10-03).

**A correction to my own earlier reasoning, recorded because it was wrong in the handoff log.** The
previous entry asserted a bare-IP URL "can NEVER validate" and treated a SAN match as a
*requirement*. The operator pointed out it is not: `httpsRequired=0` and `secureConnections` default
to Preferred, so clients can fall back to cleartext. The measurements still stand (a validating
client did fail, and real clients did send `certificate unknown`) — but the correct framing is that a
matching certificate keeps the remote leg **encrypted**, not that it is required for Plex to
function. That is why this branch obtains a real certificate for the advertised name rather than
choosing the name to suit Plex's built-in one.

**SENSITIVE PATHS — touches `clusters/**` (two files), so `ci/approval-policy.sh` will BLOCK until an
approving review (or the operator's `/approve-override <reason>` posted as a PR *review* of type
"Comment"). Deliberately not self-posted: CLAUDE.md §1.**

## What I verified

Live prod, 2026-10-04, read-only. Secrets per CLAUDE.md §6: prefs were read by name-scoped grep,
the server token was used inside the pod and never printed, and no `Preferences.xml` was dumped.

- **The LAN-classification root cause, directly:** `LanNetworksBandwidth=<ABSENT>`, and request
  tagging of **468 × `Allowed Network (WAN)` vs 0 × `(LAN)`** — including `10.0.2.33` (Seerr), an
  in-cluster pod. This is the measurement the whole hook rests on.
- **All four prefs the hook sets EXIST on this build**, read from the server's own `/:/prefs`
  (token-authenticated, output filtered to ids): `LanNetworksBandwidth`, `customCertificatePath`,
  `customCertificateKey`, `customCertificateDomain`, all `type="text"`. I had earlier been unable to
  confirm this (an unauthenticated `GET /:/prefs` returns **401**) and said so rather than assuming.
- **There is NO env path for these settings** — the reason the hook exists. Grep of the image's
  entire init surface: `setPref` is called for exactly six prefs (`MachineIdentifier`,
  `ProcessedMachineIdentifier`, `PlexOnlineToken`, `customConnections`, `allowedNetworks`,
  `TranscoderTempDirectory`) and the only `PLEX_*` env it reads are CLAIM/CLAIM_FILE/UID/GID/
  UPDATE_CHANNEL and five INFO/HOME vars. No `PLEX_PREFERENCE*` of any kind; zero files mention
  `LanNetworks`.
- **The DNS gap was real:** `plex-vip.${BASE_DOMAIN}` resolved to `192.168.2.20` (Gateway VIP), and
  `https://192.168.2.20:32400` returned `http=000` (nothing listening). PiHole's wildcard is why.
- **`openssl` is present in the image** (`/usr/bin/openssl`, 3.0.13) — the hook's PEM→PKCS#12
  conversion has a tool to run.
- **The issuer exists and works:** ClusterIssuer `letsencrypt-prod` `READY=True` (28d), and the
  existing `gateway/wildcard-home` Certificate is `Ready`/not-expired, which is what makes a DNS01
  issuance for a new name in this zone a reasonable expectation.
- **The renderer's substitution guard caught a real defect in my own work**, which is recorded
  because it is the second time this class of bug has appeared here: my hook's explanatory comment
  contained brace-wrapped names, and `assert_substitution_safe` refused to emit
  (`'PX' is not a declared cluster-vars key — envsubst will replace it with the EMPTY STRING`).
  Reworded; the rendered hook now contains exactly one brace-wrapped name, `BASE_DOMAIN`, verified
  by parsing the emitted manifest.
- **A `set -u` bug I introduced and then removed.** With `optional: true` the TLS env vars are
  genuinely UNSET when no certificate has issued; under `set -u` the first `[ -n ... ]` would abort
  the hook, and a postStart hook exiting non-zero makes kubelet KILL the container — the exact
  failure `optional` exists to prevent. Verified by parsing the rendered manifest that no `set -u`
  **command** remains (only two mentions in comments).
- **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with
  `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all
  **pass**; the rc 3 is the two known environmental skips (`gitea-settings` — no local
  `SETTINGS_API_TOKEN`; `conformance` L2/L3 — `CLUSTER_AVAILABLE` unset). **Not green. Reported as
  rc 3.** `l0` passing is the specific proof the schema edit admits `optional` and still validates
  every catalog file; `render-golden` passing is the proof catalog and committed goldens agree.
- **Only plex and pihole goldens moved.** No other service's manifest changed, and no service was
  silently onboarded.

## What I did NOT verify

- **THE OUTCOME, ENTIRELY. Nothing in this branch has been observed working.** It is unreconciled:
  no Certificate exists in prod, `plex-vip` still resolves to the Gateway VIP, `LanNetworksBandwidth`
  is still absent, and the hook has NEVER RUN. Every claim above is about the inputs, not the result.
- **Whether the certificate will actually ISSUE.** DNS01 for a new name in this zone is inferred from
  the wildcard's success, not demonstrated. If the Cloudflare credential or solver is scoped in a way
  that fails for this name, Plex will start (by design) with no TLS and the advertised `https://` URL
  will be unservable — i.e. **this branch could leave the advertised URL broken in a new way**. First
  reconcile must be watched.
- **Whether `openssl pkcs12 -export` produces a bundle PLEX ACCEPTS.** OpenSSL 3 defaults to AES-256
  + SHA-256, which some consumers reject. Untested against Plex. If it is rejected, Plex may fail to
  load its TLS listener — the hook guards against a *missing* file, NOT against a present-but-
  unreadable one. **This is the single largest unverified risk in the change.**
- **Whether the certificate prefs take effect without a restart.** Plex may need to re-initialise its
  listener after `customCertificatePath` changes. If so, the first reconcile applies the prefs and
  the SECOND start serves the cert. Converges either way, but the first start may look wrong.
- **Whether `LanNetworksBandwidth` actually flips the Remote badge.** It is the correct setting by
  Plex's own documentation and the measured absence explains the symptom exactly, but I did not set it
  live and observe a client reclassify.
- **SNAT makes the LAN definition coarser than it looks.** With `externalTrafficPolicy: Cluster`, Plex
  sees the ingress NODE's address for everything arriving via the VIP, so once `10.0.0.0/8` and
  `192.168.0.0/16` are LAN, GENUINE REMOTE clients arriving that way will ALSO be classified local
  and remote bandwidth limits will not bind on them. This is the metering consequence already
  accepted on 2026-10-03, now made stronger rather than weaker. Not measured.
- **The public Cloudflare record for `plex-vip` is off-repo and absent from this branch**, as is the
  WAN tcp/32400 port-forward. Remote access cannot work until the operator creates both; nothing here
  can verify either, and this repo has no off-LAN vantage point.
- **The private key is exposed in the container's environment** (`/proc/1/environ`) because the schema
  and renderer support no secret VOLUME — measured: `storage` accepts only
  [class, export, mountPath, name, readOnly, size, subPath] with `additionalProperties: false`, and
  no rendered app mounts a Secret. A volume would be the better shape; adding secret-volume support
  is the clean follow-up and I did not do it.
- **The PKCS#12 passphrase is a hardcoded literal** (`plexvip`) in the rendered manifest. It protects
  nothing — Plex needs it in cleartext in `customCertificateKey` and it sits beside the bundle on the
  same PVC — but it is worth knowing it is not a secret and was never treated as one.
- **Four places must agree and nothing enforces it:** `ADVERTISE_IP`, the hook's
  `customCertificateDomain`, `plex-cert.yaml`'s `dnsNames`, and PiHole's answer. No gate stage or
  conformance test checks that they match, or that PiHole's `plex-vip` answer still equals
  `${PLEX_L4_VIP}`.