From 874c6703f0d10a344ba1eb14d1f4496b88a5b4e9 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Mon, 5 Oct 2026 20:58:55 -0500 Subject: [PATCH] fix(plex): advertise a resolver-independent plex.direct name again (LAN-wide outage) I took Plex down for the whole LAN. ADVERTISE_IP was http://plex-vip.${BASE_DOMAIN}:32400 -- a name that exists ONLY in PiHole's split-horizon answers and in no public zone. A Plex client resolves the advertised host ITSELF, with whatever resolver it uses, and on a real LAN that is frequently not PiHole: browsers ship DoH on by default, phones have "private DNS", guest/IoT devices take the router's DNS. Those clients could not resolve the name, the router answered the failed lookup with ITSELF, and they all landed on its web UI -- hence the %D certificate. Measured 2026-10-05: 192.168.2.1:443 -> C=ZZ, O=%D342e7e6b, CN=%D (router, LAN side) 136.51.115.252:443 -> the same %D cert (router, WAN side) pihole-0 AND pihole-1 -> plex -> .20, plex-vip -> .24 (both CORRECT) gateway .20:443 -> CN=*.home.keithjackson.dev, valid http://192.168.2.24:32400/identity -> 200 Plex's published connections: all port 32400, nothing on 443 The DNS was right; the DEPENDENCY on it was the bug. And since nothing Plex advertises listens on 443, the clients had left the Plex path entirely. Fix: ..plex.direct, which is RESOLVER-INDEPENDENT -- a public zone returning the private address embedded in the label, so every resolver (PiHole, the router, 1.1.1.1, a DoH browser) answers it with 192.168.2.24. No split-horizon entry, no Cloudflare record and no certificate work needed: the wildcard cert Plex already serves on 32400 covers that name. Re-verified today with full validation (no curl -k): 200, TLS verified. The nicer hostname cannot be made correct for clients that bypass PiHole -- a public record resolves to the WAN address and there is no NAT hairpin, so it is right only for PiHole users and wrong for the rest. There is no configuration of it that is right for both. PiHole's plex-vip answer is KEPT; it is just not what Plex advertises. Cost restated, not buried: the value embeds the VIP and the cert hash as a literal and nothing checks they still agree. Accepted because an unresolvable name is a worse failure -- silent, LAN-wide, and it points users at the router. UNVERIFIED: unreconciled, and the client-side resolver path is INFERRED -- my public-resolver probe was broken (NXDOMAIN even for a name that resolves) so I discarded it. The chain is the only explanation consistent with the measurements, but the NXDOMAIN-hijack step was not observed directly. Co-Authored-By: Claude Opus 5 (1M context) --- apps/wave-0/plex.yaml | 2 +- catalog/plex.yaml | 110 ++++++++---------- ...-05-plex-resolver-independent-advertise.md | 105 +++++++++++++++++ 3 files changed, 152 insertions(+), 65 deletions(-) create mode 100644 decisions/2026-10-05-plex-resolver-independent-advertise.md diff --git a/apps/wave-0/plex.yaml b/apps/wave-0/plex.yaml index 5165af4..8fb19f2 100644 --- a/apps/wave-0/plex.yaml +++ b/apps/wave-0/plex.yaml @@ -138,7 +138,7 @@ spec: name: plex-claim key: PLEX_CLAIM - name: ADVERTISE_IP - value: http://plex-vip.${BASE_DOMAIN}:32400 + value: https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400 - name: PLEX_TLS_CRT valueFrom: secretKeyRef: diff --git a/catalog/plex.yaml b/catalog/plex.yaml index aaea8ba..2c1521e 100644 --- a/catalog/plex.yaml +++ b/catalog/plex.yaml @@ -72,77 +72,59 @@ env: secretKeyRef: name: plex-claim key: PLEX_CLAIM - # ADVERTISE_IP ("Custom server access URLs") — the ONE hostname this server publishes to plex.tv, - # serving BOTH local and remote clients by split-horizon DNS (operator design, 2026-10-04): - # LAN -> PiHole answers plex-vip.${BASE_DOMAIN} with ${PLEX_L4_VIP} (the tcp/32400 LoadBalancer - # in apps/wave-0/plex-remote.yaml). See catalog/pihole.yaml's split-horizon line, which - # needs a per-host override because the zone wildcard otherwise points at the GATEWAY - # VIP — measured 2026-10-04: plex-vip.${BASE_DOMAIN} resolved to the Gateway, where - # nothing listens on 32400. - # PUBLIC -> Cloudflare answers with the WAN address, NATed to the same VIP (off-repo). + # ADVERTISE_IP ("Custom server access URLs") — the connection Plex publishes to plex.tv. # - # WHY A HOSTNAME AND NOT THE `plex.direct` FORM THIS BRIEFLY USED. The first fix for the - # 2026-10-04 outage advertised `https://..plex.direct:32400`, which works - # (and is Plex's own mechanism), but it hard-codes BOTH the VIP and the server's certificate hash - # into a cluster-var — two values nothing could keep in sync, with no alert if either drifted. A - # hostname in our own zone removes that whole class of problem: ${BASE_DOMAIN} is already a knob, - # the VIP lives only in PiHole's answer, and `PLEX_ADVERTISE_URL` is deleted outright. + # THE VALUE MUST BE RESOLVABLE BY EVERY CLIENT, NOT JUST BY CLIENTS THAT USE OUR DNS. That single + # property is what this field keeps getting wrong, so it is the first thing to check before + # changing it. A Plex client resolves the advertised host ITSELF, with whatever resolver it + # happens to use — and on a real LAN that is frequently NOT PiHole: browsers ship DoH on by + # default, phones have "private DNS", guest and IoT devices often take the router's DNS, and some + # clients hard-code a public resolver. # - # THIS IS `https`, WHICH ONLY WORKS BECAUSE WE HAND PLEX OUR OWN CERTIFICATE. Plex serves 32400 - # with its `*..plex.direct` cert, which does NOT cover this name — advertising - # `https://plex-vip...` against the stock cert reproduces the exact failure that caused the - # outage (measured: a storm of `CERT: incomplete TLS handshake ... certificate unknown` from real - # clients, and `curl` without `-k` failing to connect). The `lifecycle.postStart` hook below - # installs the Let's Encrypt cert for this name and points Plex at it, so the advertised URL and - # the served certificate agree. Do not change this to a bare IP, and do not drop the hook and - # leave the `https` here. + # THAT IS NOT THEORETICAL — IT TOOK PLEX DOWN FOR THE WHOLE LAN ON 2026-10-05. This field briefly + # advertised `plex-vip.${BASE_DOMAIN}`, a name that exists ONLY in PiHole's split-horizon answers + # (catalog/pihole.yaml) and in no public zone. Clients that did not use PiHole could not resolve + # it; the SOHO router answered the failed lookup with ITSELF, and every one of them landed on the + # router's own web UI and reported a certificate error for `CN = %D`. Measured 2026-10-05: the + # router serves that self-signed cert (`C = ZZ, O = %D342e7e6b, CN = %D`) on BOTH `192.168.2.1:443` + # and the WAN address, while PiHole itself was answering perfectly — both replicas returned + # `plex-vip -> ${PLEX_L4_VIP}` and `plex -> ${GATEWAY_VIP}`. The DNS was right; the DEPENDENCY on + # it was the bug. Nothing Plex advertises listens on 443, which is how we know the clients had left + # the Plex path entirely and were talking to the router. # - # (A SAN match is not what makes Plex *work* — `httpsRequired=0` and `secureConnections` default - # to Preferred, so clients can fall back to cleartext. It is what keeps the remote leg ENCRYPTED - # instead of silently downgrading traffic that crosses the internet.) + # SO: `..plex.direct`, WHICH IS RESOLVER-INDEPENDENT. This is Plex's own + # mechanism and the reason it exists: `plex.direct` is a PUBLIC zone that returns the private + # address embedded in the label, so EVERY resolver on earth — PiHole, the router, 1.1.1.1, a DoH + # browser — answers `192-168-2-24..plex.direct` with `192.168.2.24`. It needs no + # split-horizon entry, no Cloudflare record, and no certificate work, because the wildcard cert + # Plex already serves on 32400 covers exactly this name. Re-verified live 2026-10-05 with FULL + # validation (no `curl -k`): resolves to 192.168.2.24 and `/identity` returns **200, TLS verified**. + # + # WHY NOT THE NICER HOSTNAME, since a readable name in our own zone is obviously more pleasant: + # it cannot be made to work for clients that bypass PiHole. A public record for it would resolve + # to the WAN address, and with no NAT hairpin on this router a LAN client sent there fails too — + # so the hostname is correct ONLY for clients using PiHole, and wrong for the rest. There is no + # configuration of that name that is right for both. `plex.direct` is right for both. + # (The PiHole `plex-vip` answer is KEPT — it is harmless, costs nothing, and is genuinely useful + # for reaching the VIP by name from a LAN shell. It is simply not what Plex advertises.) + # + # THE TWO EMBEDDED VALUES, and the honest cost of this choice. The dashed IP must track + # ${PLEX_L4_VIP} and `` is this server's certificate identity — read it from the live + # listener with + # openssl s_client -connect :32400 .plex.direct` cert, every - # TLS-validating client rejects the hostname, and plex.tv can publish nothing. `http` works TODAY - # and is MEASURED to: PiHole answers plex-vip.${BASE_DOMAIN} with ${PLEX_L4_VIP} (verified live - # against pihole-0, 2026-10-04) and Plex serves 32400 there. - # - # THE COST, STATED: the remote leg is unencrypted. That is tolerable only because remote access - # does not work yet at all (the WAN tcp/32400 forward is still absent), so today this carries LAN - # traffic only. RESTORE `https` IN THE SAME COMMIT that re-applies the Certificate — the two are - # one change, and splitting them is what broke this. - # - # HOW TO GET `https` BACK. The Certificate that would serve this name was applied on 2026-10-04 - # and STALLED THE ENTIRE APP TIER; the file was DELETED rather than merely unregistered, because - # `tls-lifecycle-automated` (R-TLS-02) fails any Certificate left in the tree that no Flux path - # reconciles. Full post-mortem: decisions/2026-10-04-plex-unblock-app-tier.md. Three things must - # land TOGETHER: - # 1. EXEMPT cert-manager FROM `secret-provenance`. Its temporary private-key Secrets - # (`plex-vip-`) are denied by `secrets-must-be-flux-managed`, which matches every - # Secret in a namespace labelled `homelab.keithjackson.dev/app-namespace: true` — `plex` HAS - # that label, `gateway` (where the working wildcard cert lives) does NOT, which is why this - # never bit the existing Certificate. `secretTemplate` cannot fix it: those Secrets come from - # the keymanager controller, not the Certificate's template. Narrow fix: an - # `exclude.any[].subjects` entry for ServiceAccount `cert-manager`/`cert-manager` (confirmed - # live as the controller's SA). `platform/kyverno/**` is a SENSITIVE path — non-author approval. - # 2. GIVE THE CERTIFICATE A NON-GATING HOME. `apps-wave-0` is `wait: true`, so Flux health-checks - # the Certificate OBJECT and a pending issuance stalls the whole tier - # (`timeout waiting for: [Certificate/plex/plex-vip status: 'InProgress']`). `optional: true` - # on the env refs below protects the POD, not the Kustomization — that distinction is the - # whole bug. Use a separate Kustomization with `wait: false`, the shape - # clusters/base/backup.yaml already uses for exactly this reason. - # 3. Flip this value back to `https`. + # PLEX_CLAIM/PlexOnlineToken guard, with its `/.firstRunComplete` marker on the container OVERLAY + # fs rather than the /config PVC — so this is re-applied on EVERY pod start and git, not a web + # form, owns the value. - name: ADVERTISE_IP - value: http://plex-vip.${BASE_DOMAIN}:32400 + value: https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400 # THE TLS MATERIAL FOR THE NAME ABOVE, as PEM text in env rather than a mounted volume. # `plex-vip-tls` is issued by cert-manager from ClusterIssuer `letsencrypt-prod` for exactly # `plex-vip.${BASE_DOMAIN}` (apps/wave-0/plex-cert.yaml, same wave => same reconcile as this diff --git a/decisions/2026-10-05-plex-resolver-independent-advertise.md b/decisions/2026-10-05-plex-resolver-independent-advertise.md new file mode 100644 index 0000000..efe4482 --- /dev/null +++ b/decisions/2026-10-05-plex-resolver-independent-advertise.md @@ -0,0 +1,105 @@ +--- +date: 2026-10-05 +branch: fix/plex-resolver-independent-advertise +title: Plex down LAN-wide — the advertised name existed only in PiHole, so clients that bypass it landed on the router's %D cert +--- + +## What changed + +**I took Plex down for the whole LAN.** Reported symptom: every internal client showed a certificate +error with `%D` as the subject name and could not connect. + +**Root cause: the advertised connection depended on our own resolver.** `ADVERTISE_IP` was +`http://plex-vip.${BASE_DOMAIN}:32400` — a name that exists ONLY in PiHole's split-horizon answers +and in no public zone. A Plex client resolves the advertised host *itself*, with whatever resolver it +happens to use, and on a real LAN that is frequently not PiHole: browsers ship DoH on by default, +phones have "private DNS", guest/IoT devices take the router's DNS. Those clients could not resolve +the name, the SOHO router answered the failed lookup with **itself**, and they all landed on the +router's web UI. + +| Measured 2026-10-05 | Result | +|---|---| +| `192.168.2.1:443` (router, LAN side) | `C = ZZ, ST = Somewhere, O = %D342e7e6b, CN = %D` | +| `136.51.115.252:443` (router, WAN side) | the same `%D` cert | +| PiHole **pihole-0** | `plex -> 192.168.2.20`, `plex-vip -> 192.168.2.24` ✅ | +| PiHole **pihole-1** | identical ✅ (both behind the `.23` VIP) | +| Gateway `.20:443` cert | `CN = *.home.keithjackson.dev`, valid to 2026-12-12 ✅ | +| `http://192.168.2.24:32400/identity` | `200` ✅ | +| Plex's published connections | all port **32400** — nothing on 443 | + +**The DNS was right; the dependency on it was the bug.** PiHole answered correctly from both +replicas the entire time. And because nothing Plex advertises listens on 443, the clients had plainly +left the Plex path altogether and were talking to the router. + +**The fix: go back to `..plex.direct`, which is RESOLVER-INDEPENDENT.** This +is Plex's own mechanism and the reason it exists — `plex.direct` is a public zone that returns the +private address embedded in the label, so *every* resolver (PiHole, the router, 1.1.1.1, a DoH +browser) answers `192-168-2-24..plex.direct` with `192.168.2.24`. It needs no split-horizon +entry, no Cloudflare record and no certificate work, because the wildcard cert Plex already serves on +32400 covers exactly that name. + +- **`catalog/plex.yaml`:** `ADVERTISE_IP` → + `https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400`, with the + resolver-independence requirement written at the top of the block as the first thing to check + before anyone changes this field again. +- **PiHole's `plex-vip` answer is KEPT.** It is harmless, costs nothing, and is useful for reaching + the VIP by name from a LAN shell. It is simply not what Plex advertises. + +**Why not keep the nicer hostname.** It cannot be made correct for clients that bypass PiHole: a +public record for it resolves to the WAN address, and with no NAT hairpin on this router a LAN client +sent there fails too. The name is right only for PiHole users and wrong for everyone else, and there +is no configuration of it that is right for both. `plex.direct` is right for both. + +**The honest cost, restated rather than buried:** the value embeds both the VIP and the server's +certificate hash as a literal — envsubst cannot dash an IP and nothing in the tree can derive the +hash, so nothing checks that they still agree. That fragility is exactly why the hostname was tried. +It is accepted because an unresolvable name is a *worse* failure: silent, LAN-wide, and it points +users at their router. + +## What I verified + +Live prod, 2026-10-05, read-only. + +- **Every row of the table above**, including the router's `%D` cert on both its LAN and WAN sides — + which is what identifies the router as what the clients were reaching. +- **THE FIX ITSELF, with full TLS validation (no `curl -k`)**: the plex.direct name resolves to + `192.168.2.24` and `/identity` returns **200, TLS verified**. Re-checked today, not carried over + from yesterday's measurement. +- **Both PiHole replicas are identical and correct**, with the same five `address=` lines live in + `/etc/pihole/dnsmasq.conf`, and both IPs present behind the `pihole-dns-tcp`/`-udp` endpoints. I + checked only pihole-0 at first and deliberately went back for pihole-1, because a split-brain pair + would have produced exactly this intermittent-looking symptom. +- **The cluster was never the problem:** all eight LoadBalancer VIPs still assigned, L2 + announcements intact (prod-3 holds `.20`/`.23`/`.24`), `wildcard-home` Certificate `Ready=True`, + and PR #51 had reconciled (`customConnections="http://plex-vip.home.keithjackson.dev:32400"` was + live, confirming the bad value was in effect). +- **`LanNetworksBandwidth="10.0.0.0/8,192.168.0.0/16"` is still set**, so the postStart hook survived + the roll and keeps re-asserting. +- **The Cilium rollout from #50 STILL has not happened** — agents are unchanged since + 2026-10-04T07:11, so the Hubble export is inactive and `filePath: stdout` remains unproven. + +## What I did NOT verify + +- **THE OUTCOME. Unreconciled.** No client has been observed connecting after this change. The + argument is that the advertised name now resolves for every resolver and serves a matching + certificate — both measured — but I have not seen a Plex client complete a connection, and I + cannot see the user's clients at all. +- **THE CLIENT-SIDE RESOLVER PATH IS INFERRED, NOT OBSERVED.** I could not query a public resolver + from here — my attempt returned NXDOMAIN even for a `plex.direct` name that certainly resolves, so + the probe was broken (egress policy or a missing tool) and I discarded it rather than report it. + The chain "client bypasses PiHole → NXDOMAIN → router answers with itself → `%D`" is the only + explanation consistent with all the measurements, but the NXDOMAIN-hijack step in particular is + deduced from the router serving `%D` on its LAN address, not from watching a lookup. +- **Which clients bypass PiHole, or how many.** If in fact every client uses PiHole, then my + explanation is wrong and something else sent them to the router — in which case the next thing to + check is whether the clients were using `plex.home.keithjackson.dev` (port 443, the Gateway) rather + than an advertised connection at all. +- **Whether plex.tv will re-publish promptly** after the value changes, and whether stale connection + lists are cached on clients. A client holding the old list may keep failing until it refreshes. +- **Mixed content.** The previous `http://` value would also have been blocked by the browser for + anyone using `https://app.plex.tv`; that is fixed incidentally by returning to `https`, but I did + not test it. +- **The Remote badge — the symptom that began this whole sequence — remains unconfirmed.** + `LanNetworksBandwidth` is set, but Plex still tags traffic `Allowed Network (WAN)`. Unchanged and + still unexplained. +- **`ManualPortMappingMode="1"` with an absent port** is still there, untouched. -- 2.54.0