Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

fix(plex): advertise a resolver-independent plex.direct name again (LAN-wide outage) #52

Merged
jakman85 merged 1 commits from fix/plex-resolver-independent-advertise into main 2026-10-06 02:06:00 +00:00
Conversation 0 Commits 1 Files Changed 3
+152 -65
gitea-admin commented 2026-10-06 01:59:24 +00:00
Owner
Copy link
Copy source

I took Plex down for the whole LAN. This restores it. Merge and reconcile this one first.

Cause

ADVERTISE_IP was http://plex-vip.${BASE_DOMAIN}:32400 — a name that exists only in PiHole's
split-horizon answers and in no public zone.

A Plex client resolves the advertised host itself, with whatever resolver it happens to use — and
on a real LAN that is frequently not PiHole: browsers ship DoH on by default, phones have "private
DNS", guest and IoT devices take the router's DNS. Those clients couldn't resolve the name, the
router answered the failed lookup with itself, and they all landed on its web UI — hence %D.

Measured 2026-10-05 Result
192.168.2.1:443 (router, LAN side) C = ZZ, O = %D342e7e6b, CN = %D
136.51.115.252:443 (router, WAN side) the same %D cert
PiHole pihole-0 plex → .20, plex-vip → .24 ✅
PiHole pihole-1 identical ✅ (both behind the .23 VIP)
Gateway .20:443 CN = *.home.keithjackson.dev, valid to Dec 12 ✅
http://192.168.2.24:32400/identity 200 ✅
Plex's published connections all port 32400 — nothing on 443

The DNS was right; the dependency on it was the bug. PiHole answered correctly from both replicas
the whole time, and since nothing Plex advertises listens on 443, the clients had clearly left the
Plex path entirely and were talking to the router.

Fix

Back to <dashed-ip>.<serverhash>.plex.direct, which is resolver-independent — Plex's own
mechanism, and the reason it exists. plex.direct is a public zone that returns the private address
embedded in the label, so every resolver (PiHole, the router, 1.1.1.1, a DoH browser) answers
192-168-2-24.<hash>.plex.direct with 192.168.2.24. No split-horizon entry, no Cloudflare record
and no certificate work, because the wildcard cert Plex already serves on 32400 covers that exact
name.

Re-verified today with full validation (no curl -k): resolves to 192.168.2.24, /identity
returns 200, TLS verified.

PiHole's plex-vip answer is kept — harmless, and useful for reaching the VIP by name from a LAN
shell. It's just not what Plex advertises.

Why not keep the nicer hostname

It can't be made correct for clients that bypass PiHole. A public record for it resolves to the WAN
address, and with no NAT hairpin a LAN client sent there fails too. So the name is right only for
PiHole users and wrong for everyone else — there is no configuration of it that is right for both.
plex.direct is right for both.

The requirement is now written at the top of that block as the first thing to check before anyone
changes the field again.

The cost, restated rather than buried

The value embeds both the VIP and the server's cert hash as a literal — envsubst can't dash an IP and
nothing in the tree can derive the hash, so nothing checks they still agree. That fragility is
exactly why the hostname was tried. It's accepted because an unresolvable name is a worse failure:
silent, LAN-wide, and it points users at their router.

Gate

rc 3 — spec-immutability, decisions-log, lint, l0, render-golden, l1,
approval-policy all pass; rc 3 is the two known environmental skips. Not green — reported as
rc 3. No sensitive paths.

Unverified

  • Unreconciled — no client observed connecting yet.
  • The client-side resolver path is INFERRED, not observed. My public-resolver probe was broken
    (NXDOMAIN even for a plex.direct name that certainly resolves), so I discarded it. The chain
    "bypasses PiHole → NXDOMAIN → router answers with itself → %D" is the only explanation consistent
    with every measurement, but the hijack step was deduced from the router serving %D on its LAN
    address, not from watching a lookup. If in fact every client uses PiHole then my explanation is
    wrong, and the next thing to check is whether clients were hitting
    plex.home.keithjackson.dev (443, the Gateway) rather than an advertised connection at all.
  • Clients holding a stale connection list may keep failing until they refresh from plex.tv.
  • The Remote badge — the symptom that started all this — is still unconfirmed and unexplained.
  • The Cilium rollout from #50 still hasn't happened, so the Hubble export is inactive and
    filePath: stdout remains unproven.

🤖 Generated with Claude Code

**I took Plex down for the whole LAN. This restores it.** Merge and reconcile this one first. ## Cause `ADVERTISE_IP` was `http://plex-vip.${BASE_DOMAIN}:32400` — **a name that exists only in PiHole's split-horizon answers and in no public zone.** A Plex client resolves the advertised host *itself*, with whatever resolver it happens to use — and on a real LAN that is frequently not PiHole: browsers ship DoH on by default, phones have "private DNS", guest and IoT devices take the router's DNS. Those clients couldn't resolve the name, the router answered the failed lookup with **itself**, and they all landed on its web UI — hence `%D`. | Measured 2026-10-05 | Result | |---|---| | `192.168.2.1:443` (router, **LAN** side) | `C = ZZ, O = %D342e7e6b, CN = %D` | | `136.51.115.252:443` (router, WAN side) | the same `%D` cert | | PiHole **pihole-0** | `plex → .20`, `plex-vip → .24` ✅ | | PiHole **pihole-1** | identical ✅ (both behind the `.23` VIP) | | Gateway `.20:443` | `CN = *.home.keithjackson.dev`, valid to Dec 12 ✅ | | `http://192.168.2.24:32400/identity` | `200` ✅ | | Plex's published connections | all port **32400** — nothing on 443 | **The DNS was right; the dependency on it was the bug.** PiHole answered correctly from both replicas the whole time, and since nothing Plex advertises listens on 443, the clients had clearly left the Plex path entirely and were talking to the router. ## Fix Back to `<dashed-ip>.<serverhash>.plex.direct`, which is **resolver-independent** — Plex's own mechanism, and the reason it exists. `plex.direct` is a public zone that returns the private address embedded in the label, so *every* resolver (PiHole, the router, 1.1.1.1, a DoH browser) answers `192-168-2-24.<hash>.plex.direct` with `192.168.2.24`. No split-horizon entry, no Cloudflare record and no certificate work, because the wildcard cert Plex already serves on 32400 covers that exact name. Re-verified today with full validation (no `curl -k`): resolves to `192.168.2.24`, `/identity` returns **200, TLS verified**. **PiHole's `plex-vip` answer is kept** — harmless, and useful for reaching the VIP by name from a LAN shell. It's just not what Plex advertises. ## Why not keep the nicer hostname It can't be made correct for clients that bypass PiHole. A public record for it resolves to the WAN address, and with no NAT hairpin a LAN client sent there fails too. So the name is right only for PiHole users and wrong for everyone else — there is no configuration of it that is right for both. `plex.direct` is right for both. The requirement is now written at the top of that block as the first thing to check before anyone changes the field again. ## The cost, restated rather than buried The value embeds both the VIP and the server's cert hash as a literal — envsubst can't dash an IP and nothing in the tree can derive the hash, so nothing checks they still agree. That fragility is exactly why the hostname was tried. It's accepted because an unresolvable name is a *worse* failure: silent, LAN-wide, and it points users at their router. ## Gate **rc 3** — `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1`, `approval-policy` all pass; rc 3 is the two known environmental skips. **Not green — reported as rc 3.** No sensitive paths. ## Unverified - **Unreconciled** — no client observed connecting yet. - **The client-side resolver path is INFERRED, not observed.** My public-resolver probe was broken (NXDOMAIN even for a `plex.direct` name that certainly resolves), so I discarded it. The chain "bypasses PiHole → NXDOMAIN → router answers with itself → `%D`" is the only explanation consistent with every measurement, but the hijack step was deduced from the router serving `%D` on its LAN address, not from watching a lookup. If in fact *every* client uses PiHole then my explanation is wrong, and the next thing to check is whether clients were hitting `plex.home.keithjackson.dev` (443, the Gateway) rather than an advertised connection at all. - Clients holding a stale connection list may keep failing until they refresh from plex.tv. - **The Remote badge — the symptom that started all this — is still unconfirmed and unexplained.** - **The Cilium rollout from #50 still hasn't happened**, so the Hubble export is inactive and `filePath: stdout` remains unproven. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
gitea-admin added 1 commit 2026-10-06 01:59:24 +00:00
fix(plex): advertise a resolver-independent plex.direct name again (LAN-wide outage)
gate / gate (pull_request) Successful in 1m50s
Details
gate / conformance (pull_request) Has been skipped
Details
874c6703f0 
I took Plex down for the whole LAN. ADVERTISE_IP was
http://plex-vip.${BASE_DOMAIN}:32400 -- a name that exists ONLY in PiHole's
split-horizon answers and in no public zone.

A Plex client resolves the advertised host ITSELF, with whatever resolver it
uses, and on a real LAN that is frequently not PiHole: browsers ship DoH on by
default, phones have "private DNS", guest/IoT devices take the router's DNS.
Those clients could not resolve the name, the router answered the failed lookup
with ITSELF, and they all landed on its web UI -- hence the %D certificate.

Measured 2026-10-05:
  192.168.2.1:443      -> C=ZZ, O=%D342e7e6b, CN=%D   (router, LAN side)
  136.51.115.252:443   -> the same %D cert            (router, WAN side)
  pihole-0 AND pihole-1 -> plex -> .20, plex-vip -> .24   (both CORRECT)
  gateway .20:443      -> CN=*.home.keithjackson.dev, valid
  http://192.168.2.24:32400/identity -> 200
  Plex's published connections: all port 32400, nothing on 443

The DNS was right; the DEPENDENCY on it was the bug. And since nothing Plex
advertises listens on 443, the clients had left the Plex path entirely.

Fix: <dashed-ip>.<serverhash>.plex.direct, which is RESOLVER-INDEPENDENT -- a
public zone returning the private address embedded in the label, so every
resolver (PiHole, the router, 1.1.1.1, a DoH browser) answers it with
192.168.2.24. No split-horizon entry, no Cloudflare record and no certificate
work needed: the wildcard cert Plex already serves on 32400 covers that name.
Re-verified today with full validation (no curl -k): 200, TLS verified.

The nicer hostname cannot be made correct for clients that bypass PiHole -- a
public record resolves to the WAN address and there is no NAT hairpin, so it is
right only for PiHole users and wrong for the rest. There is no configuration of
it that is right for both. PiHole's plex-vip answer is KEPT; it is just not what
Plex advertises.

Cost restated, not buried: the value embeds the VIP and the cert hash as a
literal and nothing checks they still agree. Accepted because an unresolvable
name is a worse failure -- silent, LAN-wide, and it points users at the router.

UNVERIFIED: unreconciled, and the client-side resolver path is INFERRED -- my
public-resolver probe was broken (NXDOMAIN even for a name that resolves) so I
discarded it. The chain is the only explanation consistent with the
measurements, but the NXDOMAIN-hijack step was not observed directly.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
jakman85 merged commit 4386d1453b into main 2026-10-06 02:06:00 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
agent
automated-update
blast-radius
breaking
ci-tooling
dependencies
needs-review
own-image
range-widen
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
gitea-admin jakman85 (Keith Jackson) pm-agent renovate-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Home-Network/home-network-k3s-impl#52