home-network-k3s-impl/decisions/2026-10-04-plex-vip-hostname-and-lan-networks.md

date, branch, title

date	branch	title
2026-10-04	feat/plex-vip-hostname-and-lan-networks	Plex — one advertised hostname for local and remote, its own LE certificate, and a LAN definition so LAN clients stop being reported as remote

What changed

Two operator-reported symptoms, after the 2026-10-04 "server missing from the account" fix (decisions/2026-10-04-plex-advertise-plexdirect.md) had restored publishing:

1. LAN clients were shown as Remote. Root cause, measured: LanNetworksBandwidth was ABSENT, so Plex had no definition of a local network and classified everything as WAN — 468 requests logged Allowed Network (WAN) and ZERO (LAN), including Seerr dialling from inside the cluster. allowedNetworks (the ALLOWED_NETWORKS env) only waives auth; it does not make a client local. Two different settings, and only one of them has an env hook.
2. The advertised URL was a single-purpose, fragile value. The previous fix advertised https://<dashed-ip>.<serverhash>.plex.direct:32400, which works, but hard-codes the VIP and the server's certificate hash into a cluster-var — two values nothing could keep in sync.

Operator design adopted (2026-10-04): ONE hostname, plex-vip.${BASE_DOMAIN}, serving both audiences by split-horizon DNS — PiHole answers it with ${PLEX_L4_VIP} on the LAN, Cloudflare answers it publicly with the WAN address, NATed to the same VIP.

• catalog/plex.yaml: ADVERTISE_IP → https://plex-vip.${BASE_DOMAIN}:32400. New lifecycle.postStart hook (level-triggered, same idiom as pihole's and qbittorrent's) that re-asserts on every start: LanNetworksBandwidth, and the three customCertificate* prefs after converting the PEM to the PKCS#12 bundle Plex requires. New PLEX_TLS_CRT/PLEX_TLS_KEY env from the plex-vip-tls Secret, both optional: true.
• apps/wave-0/plex-cert.yaml (new, hand-written): a cert-manager Certificate for exactly plex-vip.${BASE_DOMAIN} from ClusterIssuer letsencrypt-prod. Non-catalog because the schema has no field for a Certificate. A single name, not the wildcard: the wildcard's Secret lives in gateway and Secrets are not readable across namespaces, and re-requesting *.${BASE_DOMAIN} would burn a duplicate-certificate slot for no benefit.
• schemas/servicedefinition.schema.json → 1.11.0: adds optional to env.valueFrom.secretKeyRef / configMapKeyRef. This is load-bearing, not tidiness. The Certificate must be issued into the plex namespace, and that namespace is created by the rendered apps/wave-0/plex.yaml itself — so it cannot be issued in an earlier tier and lands in the same reconcile as the workload. With hard refs, Plex would sit in CreateContainerConfigError until a certificate issued, and apps-wave-0 is wait: true with apps-wave-1 dependsOn it, so a slow or failed DNS01 challenge would have stalled the whole app tier — the hazard catalog/plex.yaml already documents for its GPU resource.
• catalog/pihole.yaml: address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP} added to FTLCONF_misc_dnsmasq_lines. Required, not cosmetic: measured 2026-10-04, plex-vip.${BASE_DOMAIN} resolved to 192.168.2.20 (the Gateway VIP) via the zone wildcard, and a connect to :32400 there returned http=000.
• clusters/base/cluster-vars.yaml + clusters/prod/kustomization.yaml: PLEX_ADVERTISE_URL deleted — the hostname makes it unnecessary, which removes the cert-hash fragility outright. PLEX_L4_VIP's note now records its second consumer (pihole).
• catalog/render.py: plex-cert EXTRA_UNITS entry; apps/wave-0/kustomization.yaml: plex-cert.yaml hand-added (NOT via --emit-waves, which would also onboard keycloak and vm-demo — the pre-existing drift recorded on 2026-10-03).

A correction to my own earlier reasoning, recorded because it was wrong in the handoff log. The previous entry asserted a bare-IP URL "can NEVER validate" and treated a SAN match as a requirement. The operator pointed out it is not: httpsRequired=0 and secureConnections default to Preferred, so clients can fall back to cleartext. The measurements still stand (a validating client did fail, and real clients did send certificate unknown) — but the correct framing is that a matching certificate keeps the remote leg encrypted, not that it is required for Plex to function. That is why this branch obtains a real certificate for the advertised name rather than choosing the name to suit Plex's built-in one.

SENSITIVE PATHS — touches clusters/** (two files), so ci/approval-policy.sh will BLOCK until an approving review (or the operator's /approve-override <reason> posted as a PR review of type "Comment"). Deliberately not self-posted: CLAUDE.md §1.

What I verified

Live prod, 2026-10-04, read-only. Secrets per CLAUDE.md §6: prefs were read by name-scoped grep, the server token was used inside the pod and never printed, and no Preferences.xml was dumped.

• The LAN-classification root cause, directly: LanNetworksBandwidth=<ABSENT>, and request tagging of 468 × Allowed Network (WAN) vs 0 × (LAN) — including 10.0.2.33 (Seerr), an in-cluster pod. This is the measurement the whole hook rests on.
• All four prefs the hook sets EXIST on this build, read from the server's own /:/prefs (token-authenticated, output filtered to ids): LanNetworksBandwidth, customCertificatePath, customCertificateKey, customCertificateDomain, all type="text". I had earlier been unable to confirm this (an unauthenticated GET /:/prefs returns 401) and said so rather than assuming.
• There is NO env path for these settings — the reason the hook exists. Grep of the image's entire init surface: setPref is called for exactly six prefs (MachineIdentifier, ProcessedMachineIdentifier, PlexOnlineToken, customConnections, allowedNetworks, TranscoderTempDirectory) and the only PLEX_* env it reads are CLAIM/CLAIM_FILE/UID/GID/ UPDATE_CHANNEL and five INFO/HOME vars. No PLEX_PREFERENCE* of any kind; zero files mention LanNetworks.
• The DNS gap was real: plex-vip.${BASE_DOMAIN} resolved to 192.168.2.20 (Gateway VIP), and https://192.168.2.20:32400 returned http=000 (nothing listening). PiHole's wildcard is why.
• openssl is present in the image (/usr/bin/openssl, 3.0.13) — the hook's PEM→PKCS#12 conversion has a tool to run.
• The issuer exists and works: ClusterIssuer letsencrypt-prod READY=True (28d), and the existing gateway/wildcard-home Certificate is Ready/not-expired, which is what makes a DNS01 issuance for a new name in this zone a reasonable expectation.
• The renderer's substitution guard caught a real defect in my own work, which is recorded because it is the second time this class of bug has appeared here: my hook's explanatory comment contained brace-wrapped names, and assert_substitution_safe refused to emit ('PX' is not a declared cluster-vars key — envsubst will replace it with the EMPTY STRING). Reworded; the rendered hook now contains exactly one brace-wrapped name, BASE_DOMAIN, verified by parsing the emitted manifest.
• A set -u bug I introduced and then removed. With optional: true the TLS env vars are genuinely UNSET when no certificate has issued; under set -u the first [ -n ... ] would abort the hook, and a postStart hook exiting non-zero makes kubelet KILL the container — the exact failure optional exists to prevent. Verified by parsing the rendered manifest that no set -u command remains (only two mentions in comments).
• Toolbox gate (./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'): rc 3, with spec-immutability, decisions-log, lint, l0, render-golden, l1 and approval-policy all pass; the rc 3 is the two known environmental skips (gitea-settings — no local SETTINGS_API_TOKEN; conformance L2/L3 — CLUSTER_AVAILABLE unset). Not green. Reported as rc 3. l0 passing is the specific proof the schema edit admits optional and still validates every catalog file; render-golden passing is the proof catalog and committed goldens agree.
• Only plex and pihole goldens moved. No other service's manifest changed, and no service was silently onboarded.

What I did NOT verify

• THE OUTCOME, ENTIRELY. Nothing in this branch has been observed working. It is unreconciled: no Certificate exists in prod, plex-vip still resolves to the Gateway VIP, LanNetworksBandwidth is still absent, and the hook has NEVER RUN. Every claim above is about the inputs, not the result.
• Whether the certificate will actually ISSUE. DNS01 for a new name in this zone is inferred from the wildcard's success, not demonstrated. If the Cloudflare credential or solver is scoped in a way that fails for this name, Plex will start (by design) with no TLS and the advertised https:// URL will be unservable — i.e. this branch could leave the advertised URL broken in a new way. First reconcile must be watched.
• Whether openssl pkcs12 -export produces a bundle PLEX ACCEPTS. OpenSSL 3 defaults to AES-256
  • SHA-256, which some consumers reject. Untested against Plex. If it is rejected, Plex may fail to load its TLS listener — the hook guards against a missing file, NOT against a present-but- unreadable one. This is the single largest unverified risk in the change.
• Whether the certificate prefs take effect without a restart. Plex may need to re-initialise its listener after customCertificatePath changes. If so, the first reconcile applies the prefs and the SECOND start serves the cert. Converges either way, but the first start may look wrong.
• Whether LanNetworksBandwidth actually flips the Remote badge. It is the correct setting by Plex's own documentation and the measured absence explains the symptom exactly, but I did not set it live and observe a client reclassify.
• SNAT makes the LAN definition coarser than it looks. With externalTrafficPolicy: Cluster, Plex sees the ingress NODE's address for everything arriving via the VIP, so once 10.0.0.0/8 and 192.168.0.0/16 are LAN, GENUINE REMOTE clients arriving that way will ALSO be classified local and remote bandwidth limits will not bind on them. This is the metering consequence already accepted on 2026-10-03, now made stronger rather than weaker. Not measured.
• The public Cloudflare record for plex-vip is off-repo and absent from this branch, as is the WAN tcp/32400 port-forward. Remote access cannot work until the operator creates both; nothing here can verify either, and this repo has no off-LAN vantage point.
• The private key is exposed in the container's environment (/proc/1/environ) because the schema and renderer support no secret VOLUME — measured: storage accepts only [class, export, mountPath, name, readOnly, size, subPath] with additionalProperties: false, and no rendered app mounts a Secret. A volume would be the better shape; adding secret-volume support is the clean follow-up and I did not do it.
• The PKCS#12 passphrase is a hardcoded literal (plexvip) in the rendered manifest. It protects nothing — Plex needs it in cleartext in customCertificateKey and it sits beside the bundle on the same PVC — but it is worth knowing it is not a secret and was never treated as one.
• Four places must agree and nothing enforces it: ADVERTISE_IP, the hook's customCertificateDomain, plex-cert.yaml's dnsNames, and PiHole's answer. No gate stage or conformance test checks that they match, or that PiHole's plex-vip answer still equals ${PLEX_L4_VIP}.