--- date: 2026-10-05 branch: fix/plex-resolver-independent-advertise title: Plex down LAN-wide — the advertised name existed only in PiHole, so clients that bypass it landed on the router's %D cert --- ## What changed **I took Plex down for the whole LAN.** Reported symptom: every internal client showed a certificate error with `%D` as the subject name and could not connect. **Root cause: the advertised connection depended on our own resolver.** `ADVERTISE_IP` was `http://plex-vip.${BASE_DOMAIN}:32400` — a name that exists ONLY in PiHole's split-horizon answers and in no public zone. A Plex client resolves the advertised host *itself*, with whatever resolver it happens to use, and on a real LAN that is frequently not PiHole: browsers ship DoH on by default, phones have "private DNS", guest/IoT devices take the router's DNS. Those clients could not resolve the name, the SOHO router answered the failed lookup with **itself**, and they all landed on the router's web UI. | Measured 2026-10-05 | Result | |---|---| | `192.168.2.1:443` (router, LAN side) | `C = ZZ, ST = Somewhere, O = %D342e7e6b, CN = %D` | | `136.51.115.252:443` (router, WAN side) | the same `%D` cert | | PiHole **pihole-0** | `plex -> 192.168.2.20`, `plex-vip -> 192.168.2.24` ✅ | | PiHole **pihole-1** | identical ✅ (both behind the `.23` VIP) | | Gateway `.20:443` cert | `CN = *.home.keithjackson.dev`, valid to 2026-12-12 ✅ | | `http://192.168.2.24:32400/identity` | `200` ✅ | | Plex's published connections | all port **32400** — nothing on 443 | **The DNS was right; the dependency on it was the bug.** PiHole answered correctly from both replicas the entire time. And because nothing Plex advertises listens on 443, the clients had plainly left the Plex path altogether and were talking to the router. **The fix: go back to `..plex.direct`, which is RESOLVER-INDEPENDENT.** This is Plex's own mechanism and the reason it exists — `plex.direct` is a public zone that returns the private address embedded in the label, so *every* resolver (PiHole, the router, 1.1.1.1, a DoH browser) answers `192-168-2-24..plex.direct` with `192.168.2.24`. It needs no split-horizon entry, no Cloudflare record and no certificate work, because the wildcard cert Plex already serves on 32400 covers exactly that name. - **`catalog/plex.yaml`:** `ADVERTISE_IP` → `https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400`, with the resolver-independence requirement written at the top of the block as the first thing to check before anyone changes this field again. - **PiHole's `plex-vip` answer is KEPT.** It is harmless, costs nothing, and is useful for reaching the VIP by name from a LAN shell. It is simply not what Plex advertises. **Why not keep the nicer hostname.** It cannot be made correct for clients that bypass PiHole: a public record for it resolves to the WAN address, and with no NAT hairpin on this router a LAN client sent there fails too. The name is right only for PiHole users and wrong for everyone else, and there is no configuration of it that is right for both. `plex.direct` is right for both. **The honest cost, restated rather than buried:** the value embeds both the VIP and the server's certificate hash as a literal — envsubst cannot dash an IP and nothing in the tree can derive the hash, so nothing checks that they still agree. That fragility is exactly why the hostname was tried. It is accepted because an unresolvable name is a *worse* failure: silent, LAN-wide, and it points users at their router. ## What I verified Live prod, 2026-10-05, read-only. - **Every row of the table above**, including the router's `%D` cert on both its LAN and WAN sides — which is what identifies the router as what the clients were reaching. - **THE FIX ITSELF, with full TLS validation (no `curl -k`)**: the plex.direct name resolves to `192.168.2.24` and `/identity` returns **200, TLS verified**. Re-checked today, not carried over from yesterday's measurement. - **Both PiHole replicas are identical and correct**, with the same five `address=` lines live in `/etc/pihole/dnsmasq.conf`, and both IPs present behind the `pihole-dns-tcp`/`-udp` endpoints. I checked only pihole-0 at first and deliberately went back for pihole-1, because a split-brain pair would have produced exactly this intermittent-looking symptom. - **The cluster was never the problem:** all eight LoadBalancer VIPs still assigned, L2 announcements intact (prod-3 holds `.20`/`.23`/`.24`), `wildcard-home` Certificate `Ready=True`, and PR #51 had reconciled (`customConnections="http://plex-vip.home.keithjackson.dev:32400"` was live, confirming the bad value was in effect). - **`LanNetworksBandwidth="10.0.0.0/8,192.168.0.0/16"` is still set**, so the postStart hook survived the roll and keeps re-asserting. - **The Cilium rollout from #50 STILL has not happened** — agents are unchanged since 2026-10-04T07:11, so the Hubble export is inactive and `filePath: stdout` remains unproven. ## What I did NOT verify - **THE OUTCOME. Unreconciled.** No client has been observed connecting after this change. The argument is that the advertised name now resolves for every resolver and serves a matching certificate — both measured — but I have not seen a Plex client complete a connection, and I cannot see the user's clients at all. - **THE CLIENT-SIDE RESOLVER PATH IS INFERRED, NOT OBSERVED.** I could not query a public resolver from here — my attempt returned NXDOMAIN even for a `plex.direct` name that certainly resolves, so the probe was broken (egress policy or a missing tool) and I discarded it rather than report it. The chain "client bypasses PiHole → NXDOMAIN → router answers with itself → `%D`" is the only explanation consistent with all the measurements, but the NXDOMAIN-hijack step in particular is deduced from the router serving `%D` on its LAN address, not from watching a lookup. - **Which clients bypass PiHole, or how many.** If in fact every client uses PiHole, then my explanation is wrong and something else sent them to the router — in which case the next thing to check is whether the clients were using `plex.home.keithjackson.dev` (port 443, the Gateway) rather than an advertised connection at all. - **Whether plex.tv will re-publish promptly** after the value changes, and whether stale connection lists are cached on clients. A client holding the old list may keep failing until it refreshes. - **Mixed content.** The previous `http://` value would also have been blocked by the browser for anyone using `https://app.plex.tv`; that is fixed incidentally by returning to `https`, but I did not test it. - **The Remote badge — the symptom that began this whole sequence — remains unconfirmed.** `LanNetworksBandwidth` is set, but Plex still tags traffic `Allowed Network (WAN)`. Unchanged and still unexplained. - **`ManualPortMappingMode="1"` with an absent port** is still there, untouched.