home-network-k3s-impl/decisions/2026-10-05-plex-resolver-independent-advertise.md

date, branch, title

date	branch	title
2026-10-05	fix/plex-resolver-independent-advertise	Plex down LAN-wide — the advertised name existed only in PiHole, so clients that bypass it landed on the router's %D cert

What changed

I took Plex down for the whole LAN. Reported symptom: every internal client showed a certificate error with %D as the subject name and could not connect.

Root cause: the advertised connection depended on our own resolver. ADVERTISE_IP was http://plex-vip.${BASE_DOMAIN}:32400 — a name that exists ONLY in PiHole's split-horizon answers and in no public zone. A Plex client resolves the advertised host itself, with whatever resolver it happens to use, and on a real LAN that is frequently not PiHole: browsers ship DoH on by default, phones have "private DNS", guest/IoT devices take the router's DNS. Those clients could not resolve the name, the SOHO router answered the failed lookup with itself, and they all landed on the router's web UI.

Measured 2026-10-05	Result
192.168.2.1:443 (router, LAN side)	C = ZZ, ST = Somewhere, O = %D342e7e6b, CN = %D
136.51.115.252:443 (router, WAN side)	the same %D cert
PiHole pihole-0	plex -> 192.168.2.20, plex-vip -> 192.168.2.24 ✅
PiHole pihole-1	identical ✅ (both behind the .23 VIP)
Gateway .20:443 cert	CN = *.home.keithjackson.dev, valid to 2026-12-12 ✅
http://192.168.2.24:32400/identity	200 ✅
Plex's published connections	all port 32400 — nothing on 443

The DNS was right; the dependency on it was the bug. PiHole answered correctly from both replicas the entire time. And because nothing Plex advertises listens on 443, the clients had plainly left the Plex path altogether and were talking to the router.

The fix: go back to <dashed-ip>.<serverhash>.plex.direct, which is RESOLVER-INDEPENDENT. This is Plex's own mechanism and the reason it exists — plex.direct is a public zone that returns the private address embedded in the label, so every resolver (PiHole, the router, 1.1.1.1, a DoH browser) answers 192-168-2-24.<hash>.plex.direct with 192.168.2.24. It needs no split-horizon entry, no Cloudflare record and no certificate work, because the wildcard cert Plex already serves on 32400 covers exactly that name.

• catalog/plex.yaml: ADVERTISE_IP → https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400, with the resolver-independence requirement written at the top of the block as the first thing to check before anyone changes this field again.
• PiHole's plex-vip answer is KEPT. It is harmless, costs nothing, and is useful for reaching the VIP by name from a LAN shell. It is simply not what Plex advertises.

Why not keep the nicer hostname. It cannot be made correct for clients that bypass PiHole: a public record for it resolves to the WAN address, and with no NAT hairpin on this router a LAN client sent there fails too. The name is right only for PiHole users and wrong for everyone else, and there is no configuration of it that is right for both. plex.direct is right for both.

The honest cost, restated rather than buried: the value embeds both the VIP and the server's certificate hash as a literal — envsubst cannot dash an IP and nothing in the tree can derive the hash, so nothing checks that they still agree. That fragility is exactly why the hostname was tried. It is accepted because an unresolvable name is a worse failure: silent, LAN-wide, and it points users at their router.

What I verified

Live prod, 2026-10-05, read-only.

• Every row of the table above, including the router's %D cert on both its LAN and WAN sides — which is what identifies the router as what the clients were reaching.
• THE FIX ITSELF, with full TLS validation (no curl -k): the plex.direct name resolves to 192.168.2.24 and /identity returns 200, TLS verified. Re-checked today, not carried over from yesterday's measurement.
• Both PiHole replicas are identical and correct, with the same five address= lines live in /etc/pihole/dnsmasq.conf, and both IPs present behind the pihole-dns-tcp/-udp endpoints. I checked only pihole-0 at first and deliberately went back for pihole-1, because a split-brain pair would have produced exactly this intermittent-looking symptom.
• The cluster was never the problem: all eight LoadBalancer VIPs still assigned, L2 announcements intact (prod-3 holds .20/.23/.24), wildcard-home Certificate Ready=True, and PR #51 had reconciled (customConnections="http://plex-vip.home.keithjackson.dev:32400" was live, confirming the bad value was in effect).
• LanNetworksBandwidth="10.0.0.0/8,192.168.0.0/16" is still set, so the postStart hook survived the roll and keeps re-asserting.
• The Cilium rollout from #50 STILL has not happened — agents are unchanged since 2026-10-04T07:11, so the Hubble export is inactive and filePath: stdout remains unproven.

What I did NOT verify

• THE OUTCOME. Unreconciled. No client has been observed connecting after this change. The argument is that the advertised name now resolves for every resolver and serves a matching certificate — both measured — but I have not seen a Plex client complete a connection, and I cannot see the user's clients at all.
• THE CLIENT-SIDE RESOLVER PATH IS INFERRED, NOT OBSERVED. I could not query a public resolver from here — my attempt returned NXDOMAIN even for a plex.direct name that certainly resolves, so the probe was broken (egress policy or a missing tool) and I discarded it rather than report it. The chain "client bypasses PiHole → NXDOMAIN → router answers with itself → %D" is the only explanation consistent with all the measurements, but the NXDOMAIN-hijack step in particular is deduced from the router serving %D on its LAN address, not from watching a lookup.
• Which clients bypass PiHole, or how many. If in fact every client uses PiHole, then my explanation is wrong and something else sent them to the router — in which case the next thing to check is whether the clients were using plex.home.keithjackson.dev (port 443, the Gateway) rather than an advertised connection at all.
• Whether plex.tv will re-publish promptly after the value changes, and whether stale connection lists are cached on clients. A client holding the old list may keep failing until it refreshes.
• Mixed content. The previous http:// value would also have been blocked by the browser for anyone using https://app.plex.tv; that is fixed incidentally by returning to https, but I did not test it.
• The Remote badge — the symptom that began this whole sequence — remains unconfirmed. LanNetworksBandwidth is set, but Plex still tags traffic Allowed Network (WAN). Unchanged and still unexplained.
• ManualPortMappingMode="1" with an absent port is still there, untouched.