diff --git a/apps/wave-0/plex.yaml b/apps/wave-0/plex.yaml index 5165af4..8fb19f2 100644 --- a/apps/wave-0/plex.yaml +++ b/apps/wave-0/plex.yaml @@ -138,7 +138,7 @@ spec: name: plex-claim key: PLEX_CLAIM - name: ADVERTISE_IP - value: http://plex-vip.${BASE_DOMAIN}:32400 + value: https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400 - name: PLEX_TLS_CRT valueFrom: secretKeyRef: diff --git a/catalog/plex.yaml b/catalog/plex.yaml index aaea8ba..2c1521e 100644 --- a/catalog/plex.yaml +++ b/catalog/plex.yaml @@ -72,77 +72,59 @@ env: secretKeyRef: name: plex-claim key: PLEX_CLAIM - # ADVERTISE_IP ("Custom server access URLs") — the ONE hostname this server publishes to plex.tv, - # serving BOTH local and remote clients by split-horizon DNS (operator design, 2026-10-04): - # LAN -> PiHole answers plex-vip.${BASE_DOMAIN} with ${PLEX_L4_VIP} (the tcp/32400 LoadBalancer - # in apps/wave-0/plex-remote.yaml). See catalog/pihole.yaml's split-horizon line, which - # needs a per-host override because the zone wildcard otherwise points at the GATEWAY - # VIP — measured 2026-10-04: plex-vip.${BASE_DOMAIN} resolved to the Gateway, where - # nothing listens on 32400. - # PUBLIC -> Cloudflare answers with the WAN address, NATed to the same VIP (off-repo). + # ADVERTISE_IP ("Custom server access URLs") — the connection Plex publishes to plex.tv. # - # WHY A HOSTNAME AND NOT THE `plex.direct` FORM THIS BRIEFLY USED. The first fix for the - # 2026-10-04 outage advertised `https://..plex.direct:32400`, which works - # (and is Plex's own mechanism), but it hard-codes BOTH the VIP and the server's certificate hash - # into a cluster-var — two values nothing could keep in sync, with no alert if either drifted. A - # hostname in our own zone removes that whole class of problem: ${BASE_DOMAIN} is already a knob, - # the VIP lives only in PiHole's answer, and `PLEX_ADVERTISE_URL` is deleted outright. + # THE VALUE MUST BE RESOLVABLE BY EVERY CLIENT, NOT JUST BY CLIENTS THAT USE OUR DNS. That single + # property is what this field keeps getting wrong, so it is the first thing to check before + # changing it. A Plex client resolves the advertised host ITSELF, with whatever resolver it + # happens to use — and on a real LAN that is frequently NOT PiHole: browsers ship DoH on by + # default, phones have "private DNS", guest and IoT devices often take the router's DNS, and some + # clients hard-code a public resolver. # - # THIS IS `https`, WHICH ONLY WORKS BECAUSE WE HAND PLEX OUR OWN CERTIFICATE. Plex serves 32400 - # with its `*..plex.direct` cert, which does NOT cover this name — advertising - # `https://plex-vip...` against the stock cert reproduces the exact failure that caused the - # outage (measured: a storm of `CERT: incomplete TLS handshake ... certificate unknown` from real - # clients, and `curl` without `-k` failing to connect). The `lifecycle.postStart` hook below - # installs the Let's Encrypt cert for this name and points Plex at it, so the advertised URL and - # the served certificate agree. Do not change this to a bare IP, and do not drop the hook and - # leave the `https` here. + # THAT IS NOT THEORETICAL — IT TOOK PLEX DOWN FOR THE WHOLE LAN ON 2026-10-05. This field briefly + # advertised `plex-vip.${BASE_DOMAIN}`, a name that exists ONLY in PiHole's split-horizon answers + # (catalog/pihole.yaml) and in no public zone. Clients that did not use PiHole could not resolve + # it; the SOHO router answered the failed lookup with ITSELF, and every one of them landed on the + # router's own web UI and reported a certificate error for `CN = %D`. Measured 2026-10-05: the + # router serves that self-signed cert (`C = ZZ, O = %D342e7e6b, CN = %D`) on BOTH `192.168.2.1:443` + # and the WAN address, while PiHole itself was answering perfectly — both replicas returned + # `plex-vip -> ${PLEX_L4_VIP}` and `plex -> ${GATEWAY_VIP}`. The DNS was right; the DEPENDENCY on + # it was the bug. Nothing Plex advertises listens on 443, which is how we know the clients had left + # the Plex path entirely and were talking to the router. # - # (A SAN match is not what makes Plex *work* — `httpsRequired=0` and `secureConnections` default - # to Preferred, so clients can fall back to cleartext. It is what keeps the remote leg ENCRYPTED - # instead of silently downgrading traffic that crosses the internet.) + # SO: `..plex.direct`, WHICH IS RESOLVER-INDEPENDENT. This is Plex's own + # mechanism and the reason it exists: `plex.direct` is a PUBLIC zone that returns the private + # address embedded in the label, so EVERY resolver on earth — PiHole, the router, 1.1.1.1, a DoH + # browser — answers `192-168-2-24..plex.direct` with `192.168.2.24`. It needs no + # split-horizon entry, no Cloudflare record, and no certificate work, because the wildcard cert + # Plex already serves on 32400 covers exactly this name. Re-verified live 2026-10-05 with FULL + # validation (no `curl -k`): resolves to 192.168.2.24 and `/identity` returns **200, TLS verified**. + # + # WHY NOT THE NICER HOSTNAME, since a readable name in our own zone is obviously more pleasant: + # it cannot be made to work for clients that bypass PiHole. A public record for it would resolve + # to the WAN address, and with no NAT hairpin on this router a LAN client sent there fails too — + # so the hostname is correct ONLY for clients using PiHole, and wrong for the rest. There is no + # configuration of that name that is right for both. `plex.direct` is right for both. + # (The PiHole `plex-vip` answer is KEPT — it is harmless, costs nothing, and is genuinely useful + # for reaching the VIP by name from a LAN shell. It is simply not what Plex advertises.) + # + # THE TWO EMBEDDED VALUES, and the honest cost of this choice. The dashed IP must track + # ${PLEX_L4_VIP} and `` is this server's certificate identity — read it from the live + # listener with + # openssl s_client -connect :32400 .plex.direct` cert, every - # TLS-validating client rejects the hostname, and plex.tv can publish nothing. `http` works TODAY - # and is MEASURED to: PiHole answers plex-vip.${BASE_DOMAIN} with ${PLEX_L4_VIP} (verified live - # against pihole-0, 2026-10-04) and Plex serves 32400 there. - # - # THE COST, STATED: the remote leg is unencrypted. That is tolerable only because remote access - # does not work yet at all (the WAN tcp/32400 forward is still absent), so today this carries LAN - # traffic only. RESTORE `https` IN THE SAME COMMIT that re-applies the Certificate — the two are - # one change, and splitting them is what broke this. - # - # HOW TO GET `https` BACK. The Certificate that would serve this name was applied on 2026-10-04 - # and STALLED THE ENTIRE APP TIER; the file was DELETED rather than merely unregistered, because - # `tls-lifecycle-automated` (R-TLS-02) fails any Certificate left in the tree that no Flux path - # reconciles. Full post-mortem: decisions/2026-10-04-plex-unblock-app-tier.md. Three things must - # land TOGETHER: - # 1. EXEMPT cert-manager FROM `secret-provenance`. Its temporary private-key Secrets - # (`plex-vip-`) are denied by `secrets-must-be-flux-managed`, which matches every - # Secret in a namespace labelled `homelab.keithjackson.dev/app-namespace: true` — `plex` HAS - # that label, `gateway` (where the working wildcard cert lives) does NOT, which is why this - # never bit the existing Certificate. `secretTemplate` cannot fix it: those Secrets come from - # the keymanager controller, not the Certificate's template. Narrow fix: an - # `exclude.any[].subjects` entry for ServiceAccount `cert-manager`/`cert-manager` (confirmed - # live as the controller's SA). `platform/kyverno/**` is a SENSITIVE path — non-author approval. - # 2. GIVE THE CERTIFICATE A NON-GATING HOME. `apps-wave-0` is `wait: true`, so Flux health-checks - # the Certificate OBJECT and a pending issuance stalls the whole tier - # (`timeout waiting for: [Certificate/plex/plex-vip status: 'InProgress']`). `optional: true` - # on the env refs below protects the POD, not the Kustomization — that distinction is the - # whole bug. Use a separate Kustomization with `wait: false`, the shape - # clusters/base/backup.yaml already uses for exactly this reason. - # 3. Flip this value back to `https`. + # PLEX_CLAIM/PlexOnlineToken guard, with its `/.firstRunComplete` marker on the container OVERLAY + # fs rather than the /config PVC — so this is re-applied on EVERY pod start and git, not a web + # form, owns the value. - name: ADVERTISE_IP - value: http://plex-vip.${BASE_DOMAIN}:32400 + value: https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400 # THE TLS MATERIAL FOR THE NAME ABOVE, as PEM text in env rather than a mounted volume. # `plex-vip-tls` is issued by cert-manager from ClusterIssuer `letsencrypt-prod` for exactly # `plex-vip.${BASE_DOMAIN}` (apps/wave-0/plex-cert.yaml, same wave => same reconcile as this diff --git a/decisions/2026-10-05-plex-resolver-independent-advertise.md b/decisions/2026-10-05-plex-resolver-independent-advertise.md new file mode 100644 index 0000000..efe4482 --- /dev/null +++ b/decisions/2026-10-05-plex-resolver-independent-advertise.md @@ -0,0 +1,105 @@ +--- +date: 2026-10-05 +branch: fix/plex-resolver-independent-advertise +title: Plex down LAN-wide — the advertised name existed only in PiHole, so clients that bypass it landed on the router's %D cert +--- + +## What changed + +**I took Plex down for the whole LAN.** Reported symptom: every internal client showed a certificate +error with `%D` as the subject name and could not connect. + +**Root cause: the advertised connection depended on our own resolver.** `ADVERTISE_IP` was +`http://plex-vip.${BASE_DOMAIN}:32400` — a name that exists ONLY in PiHole's split-horizon answers +and in no public zone. A Plex client resolves the advertised host *itself*, with whatever resolver it +happens to use, and on a real LAN that is frequently not PiHole: browsers ship DoH on by default, +phones have "private DNS", guest/IoT devices take the router's DNS. Those clients could not resolve +the name, the SOHO router answered the failed lookup with **itself**, and they all landed on the +router's web UI. + +| Measured 2026-10-05 | Result | +|---|---| +| `192.168.2.1:443` (router, LAN side) | `C = ZZ, ST = Somewhere, O = %D342e7e6b, CN = %D` | +| `136.51.115.252:443` (router, WAN side) | the same `%D` cert | +| PiHole **pihole-0** | `plex -> 192.168.2.20`, `plex-vip -> 192.168.2.24` ✅ | +| PiHole **pihole-1** | identical ✅ (both behind the `.23` VIP) | +| Gateway `.20:443` cert | `CN = *.home.keithjackson.dev`, valid to 2026-12-12 ✅ | +| `http://192.168.2.24:32400/identity` | `200` ✅ | +| Plex's published connections | all port **32400** — nothing on 443 | + +**The DNS was right; the dependency on it was the bug.** PiHole answered correctly from both +replicas the entire time. And because nothing Plex advertises listens on 443, the clients had plainly +left the Plex path altogether and were talking to the router. + +**The fix: go back to `..plex.direct`, which is RESOLVER-INDEPENDENT.** This +is Plex's own mechanism and the reason it exists — `plex.direct` is a public zone that returns the +private address embedded in the label, so *every* resolver (PiHole, the router, 1.1.1.1, a DoH +browser) answers `192-168-2-24..plex.direct` with `192.168.2.24`. It needs no split-horizon +entry, no Cloudflare record and no certificate work, because the wildcard cert Plex already serves on +32400 covers exactly that name. + +- **`catalog/plex.yaml`:** `ADVERTISE_IP` → + `https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400`, with the + resolver-independence requirement written at the top of the block as the first thing to check + before anyone changes this field again. +- **PiHole's `plex-vip` answer is KEPT.** It is harmless, costs nothing, and is useful for reaching + the VIP by name from a LAN shell. It is simply not what Plex advertises. + +**Why not keep the nicer hostname.** It cannot be made correct for clients that bypass PiHole: a +public record for it resolves to the WAN address, and with no NAT hairpin on this router a LAN client +sent there fails too. The name is right only for PiHole users and wrong for everyone else, and there +is no configuration of it that is right for both. `plex.direct` is right for both. + +**The honest cost, restated rather than buried:** the value embeds both the VIP and the server's +certificate hash as a literal — envsubst cannot dash an IP and nothing in the tree can derive the +hash, so nothing checks that they still agree. That fragility is exactly why the hostname was tried. +It is accepted because an unresolvable name is a *worse* failure: silent, LAN-wide, and it points +users at their router. + +## What I verified + +Live prod, 2026-10-05, read-only. + +- **Every row of the table above**, including the router's `%D` cert on both its LAN and WAN sides — + which is what identifies the router as what the clients were reaching. +- **THE FIX ITSELF, with full TLS validation (no `curl -k`)**: the plex.direct name resolves to + `192.168.2.24` and `/identity` returns **200, TLS verified**. Re-checked today, not carried over + from yesterday's measurement. +- **Both PiHole replicas are identical and correct**, with the same five `address=` lines live in + `/etc/pihole/dnsmasq.conf`, and both IPs present behind the `pihole-dns-tcp`/`-udp` endpoints. I + checked only pihole-0 at first and deliberately went back for pihole-1, because a split-brain pair + would have produced exactly this intermittent-looking symptom. +- **The cluster was never the problem:** all eight LoadBalancer VIPs still assigned, L2 + announcements intact (prod-3 holds `.20`/`.23`/`.24`), `wildcard-home` Certificate `Ready=True`, + and PR #51 had reconciled (`customConnections="http://plex-vip.home.keithjackson.dev:32400"` was + live, confirming the bad value was in effect). +- **`LanNetworksBandwidth="10.0.0.0/8,192.168.0.0/16"` is still set**, so the postStart hook survived + the roll and keeps re-asserting. +- **The Cilium rollout from #50 STILL has not happened** — agents are unchanged since + 2026-10-04T07:11, so the Hubble export is inactive and `filePath: stdout` remains unproven. + +## What I did NOT verify + +- **THE OUTCOME. Unreconciled.** No client has been observed connecting after this change. The + argument is that the advertised name now resolves for every resolver and serves a matching + certificate — both measured — but I have not seen a Plex client complete a connection, and I + cannot see the user's clients at all. +- **THE CLIENT-SIDE RESOLVER PATH IS INFERRED, NOT OBSERVED.** I could not query a public resolver + from here — my attempt returned NXDOMAIN even for a `plex.direct` name that certainly resolves, so + the probe was broken (egress policy or a missing tool) and I discarded it rather than report it. + The chain "client bypasses PiHole → NXDOMAIN → router answers with itself → `%D`" is the only + explanation consistent with all the measurements, but the NXDOMAIN-hijack step in particular is + deduced from the router serving `%D` on its LAN address, not from watching a lookup. +- **Which clients bypass PiHole, or how many.** If in fact every client uses PiHole, then my + explanation is wrong and something else sent them to the router — in which case the next thing to + check is whether the clients were using `plex.home.keithjackson.dev` (port 443, the Gateway) rather + than an advertised connection at all. +- **Whether plex.tv will re-publish promptly** after the value changes, and whether stale connection + lists are cached on clients. A client holding the old list may keep failing until it refreshes. +- **Mixed content.** The previous `http://` value would also have been blocked by the browser for + anyone using `https://app.plex.tv`; that is fixed incidentally by returning to `https`, but I did + not test it. +- **The Remote badge — the symptom that began this whole sequence — remains unconfirmed.** + `LanNetworksBandwidth` is set, but Plex still tags traffic `Allowed Network (WAN)`. Unchanged and + still unexplained. +- **`ManualPortMappingMode="1"` with an absent port** is still there, untouched.