Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

fix(plex): advertise a resolver-independent plex.direct name again (LAN-wide outage)
gate / gate (pull_request) Successful in 1m50s
Details
gate / conformance (pull_request) Has been skipped
Details

Browse Source 
I took Plex down for the whole LAN. ADVERTISE_IP was
http://plex-vip.${BASE_DOMAIN}:32400 -- a name that exists ONLY in PiHole's
split-horizon answers and in no public zone.

A Plex client resolves the advertised host ITSELF, with whatever resolver it
uses, and on a real LAN that is frequently not PiHole: browsers ship DoH on by
default, phones have "private DNS", guest/IoT devices take the router's DNS.
Those clients could not resolve the name, the router answered the failed lookup
with ITSELF, and they all landed on its web UI -- hence the %D certificate.

Measured 2026-10-05:
  192.168.2.1:443      -> C=ZZ, O=%D342e7e6b, CN=%D   (router, LAN side)
  136.51.115.252:443   -> the same %D cert            (router, WAN side)
  pihole-0 AND pihole-1 -> plex -> .20, plex-vip -> .24   (both CORRECT)
  gateway .20:443      -> CN=*.home.keithjackson.dev, valid
  http://192.168.2.24:32400/identity -> 200
  Plex's published connections: all port 32400, nothing on 443

The DNS was right; the DEPENDENCY on it was the bug. And since nothing Plex
advertises listens on 443, the clients had left the Plex path entirely.

Fix: <dashed-ip>.<serverhash>.plex.direct, which is RESOLVER-INDEPENDENT -- a
public zone returning the private address embedded in the label, so every
resolver (PiHole, the router, 1.1.1.1, a DoH browser) answers it with
192.168.2.24. No split-horizon entry, no Cloudflare record and no certificate
work needed: the wildcard cert Plex already serves on 32400 covers that name.
Re-verified today with full validation (no curl -k): 200, TLS verified.

The nicer hostname cannot be made correct for clients that bypass PiHole -- a
public record resolves to the WAN address and there is no NAT hairpin, so it is
right only for PiHole users and wrong for the rest. There is no configuration of
it that is right for both. PiHole's plex-vip answer is KEPT; it is just not what
Plex advertises.

Cost restated, not buried: the value embeds the VIP and the cert hash as a
literal and nothing checks they still agree. Accepted because an unresolvable
name is a worse failure -- silent, LAN-wide, and it points users at the router.

UNVERIFIED: unreconciled, and the client-side resolver path is INFERRED -- my
public-resolver probe was broken (NXDOMAIN even for a name that resolves) so I
discarded it. The chain is the only explanation consistent with the
measurements, but the NXDOMAIN-hijack step was not observed directly.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-10-05 20:58:55 -05:00
1 parent bc7095677b
commit 874c6703f0
3 files changed
+152 -65
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

decisions/2026-10-05-plex-resolver-independent-advertise.md
+105
View File
@@ -0,0 +1,105 @@
---
date: 2026-10-05
branch: fix/plex-resolver-independent-advertise
title: Plex down LAN-wide — the advertised name existed only in PiHole, so clients that bypass it landed on the router's %D cert
---
## What changed
**I took Plex down for the whole LAN.** Reported symptom: every internal client showed a certificate
error with `%D` as the subject name and could not connect.
**Root cause: the advertised connection depended on our own resolver.** `ADVERTISE_IP` was
`http://plex-vip.${BASE_DOMAIN}:32400` — a name that exists ONLY in PiHole's split-horizon answers
and in no public zone. A Plex client resolves the advertised host *itself*, with whatever resolver it
happens to use, and on a real LAN that is frequently not PiHole: browsers ship DoH on by default,
phones have "private DNS", guest/IoT devices take the router's DNS. Those clients could not resolve
the name, the SOHO router answered the failed lookup with **itself**, and they all landed on the
router's web UI.
| Measured 2026-10-05 | Result |
|---|---|
| `192.168.2.1:443` (router, LAN side) | `C = ZZ, ST = Somewhere, O = %D342e7e6b, CN = %D` |
| `136.51.115.252:443` (router, WAN side) | the same `%D` cert |
| PiHole **pihole-0** | `plex -> 192.168.2.20`, `plex-vip -> 192.168.2.24` ✅ |
| PiHole **pihole-1** | identical ✅ (both behind the `.23` VIP) |
| Gateway `.20:443` cert | `CN = *.home.keithjackson.dev`, valid to 2026-12-12 ✅ |
| `http://192.168.2.24:32400/identity` | `200` ✅ |
| Plex's published connections | all port **32400** — nothing on 443 |
**The DNS was right; the dependency on it was the bug.** PiHole answered correctly from both
replicas the entire time. And because nothing Plex advertises listens on 443, the clients had plainly
left the Plex path altogether and were talking to the router.
**The fix: go back to `<dashed-ip>.<serverhash>.plex.direct`, which is RESOLVER-INDEPENDENT.** This
is Plex's own mechanism and the reason it exists — `plex.direct` is a public zone that returns the
private address embedded in the label, so *every* resolver (PiHole, the router, 1.1.1.1, a DoH
browser) answers `192-168-2-24.<hash>.plex.direct` with `192.168.2.24`. It needs no split-horizon
entry, no Cloudflare record and no certificate work, because the wildcard cert Plex already serves on
32400 covers exactly that name.
- **`catalog/plex.yaml`:** `ADVERTISE_IP` →
`https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400`, with the
resolver-independence requirement written at the top of the block as the first thing to check
before anyone changes this field again.
- **PiHole's `plex-vip` answer is KEPT.** It is harmless, costs nothing, and is useful for reaching
the VIP by name from a LAN shell. It is simply not what Plex advertises.
**Why not keep the nicer hostname.** It cannot be made correct for clients that bypass PiHole: a
public record for it resolves to the WAN address, and with no NAT hairpin on this router a LAN client
sent there fails too. The name is right only for PiHole users and wrong for everyone else, and there
is no configuration of it that is right for both. `plex.direct` is right for both.
**The honest cost, restated rather than buried:** the value embeds both the VIP and the server's
certificate hash as a literal — envsubst cannot dash an IP and nothing in the tree can derive the
hash, so nothing checks that they still agree. That fragility is exactly why the hostname was tried.
It is accepted because an unresolvable name is a *worse* failure: silent, LAN-wide, and it points
users at their router.
## What I verified
Live prod, 2026-10-05, read-only.
- **Every row of the table above**, including the router's `%D` cert on both its LAN and WAN sides —
which is what identifies the router as what the clients were reaching.
- **THE FIX ITSELF, with full TLS validation (no `curl -k`)**: the plex.direct name resolves to
`192.168.2.24` and `/identity` returns **200, TLS verified**. Re-checked today, not carried over
from yesterday's measurement.
- **Both PiHole replicas are identical and correct**, with the same five `address=` lines live in
`/etc/pihole/dnsmasq.conf`, and both IPs present behind the `pihole-dns-tcp`/`-udp` endpoints. I
checked only pihole-0 at first and deliberately went back for pihole-1, because a split-brain pair
would have produced exactly this intermittent-looking symptom.
- **The cluster was never the problem:** all eight LoadBalancer VIPs still assigned, L2
announcements intact (prod-3 holds `.20`/`.23`/`.24`), `wildcard-home` Certificate `Ready=True`,
and PR #51 had reconciled (`customConnections="http://plex-vip.home.keithjackson.dev:32400"` was
live, confirming the bad value was in effect).
- **`LanNetworksBandwidth="10.0.0.0/8,192.168.0.0/16"` is still set**, so the postStart hook survived
the roll and keeps re-asserting.
- **The Cilium rollout from #50 STILL has not happened** — agents are unchanged since
2026-10-04T07:11, so the Hubble export is inactive and `filePath: stdout` remains unproven.
## What I did NOT verify
- **THE OUTCOME. Unreconciled.** No client has been observed connecting after this change. The
argument is that the advertised name now resolves for every resolver and serves a matching
certificate — both measured — but I have not seen a Plex client complete a connection, and I
cannot see the user's clients at all.
- **THE CLIENT-SIDE RESOLVER PATH IS INFERRED, NOT OBSERVED.** I could not query a public resolver
from here — my attempt returned NXDOMAIN even for a `plex.direct` name that certainly resolves, so
the probe was broken (egress policy or a missing tool) and I discarded it rather than report it.
The chain "client bypasses PiHole → NXDOMAIN → router answers with itself → `%D`" is the only
explanation consistent with all the measurements, but the NXDOMAIN-hijack step in particular is
deduced from the router serving `%D` on its LAN address, not from watching a lookup.
- **Which clients bypass PiHole, or how many.** If in fact every client uses PiHole, then my
explanation is wrong and something else sent them to the router — in which case the next thing to
check is whether the clients were using `plex.home.keithjackson.dev` (port 443, the Gateway) rather
than an advertised connection at all.
- **Whether plex.tv will re-publish promptly** after the value changes, and whether stale connection
lists are cached on clients. A client holding the old list may keep failing until it refreshes.
- **Mixed content.** The previous `http://` value would also have been blocked by the browser for
anyone using `https://app.plex.tv`; that is fixed incidentally by returning to `https`, but I did
not test it.
- **The Remote badge — the symptom that began this whole sequence — remains unconfirmed.**
`LanNetworksBandwidth` is set, but Plex still tags traffic `Allowed Network (WAN)`. Unchanged and
still unexplained.
- **`ManualPortMappingMode="1"` with an absent port** is still there, untouched.