Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

fix(plex): advertise a resolver-independent plex.direct name again (LAN-wide outage)
gate / gate (pull_request) Successful in 1m50s
Details
gate / conformance (pull_request) Has been skipped
Details

Browse Source 
I took Plex down for the whole LAN. ADVERTISE_IP was
http://plex-vip.${BASE_DOMAIN}:32400 -- a name that exists ONLY in PiHole's
split-horizon answers and in no public zone.

A Plex client resolves the advertised host ITSELF, with whatever resolver it
uses, and on a real LAN that is frequently not PiHole: browsers ship DoH on by
default, phones have "private DNS", guest/IoT devices take the router's DNS.
Those clients could not resolve the name, the router answered the failed lookup
with ITSELF, and they all landed on its web UI -- hence the %D certificate.

Measured 2026-10-05:
  192.168.2.1:443      -> C=ZZ, O=%D342e7e6b, CN=%D   (router, LAN side)
  136.51.115.252:443   -> the same %D cert            (router, WAN side)
  pihole-0 AND pihole-1 -> plex -> .20, plex-vip -> .24   (both CORRECT)
  gateway .20:443      -> CN=*.home.keithjackson.dev, valid
  http://192.168.2.24:32400/identity -> 200
  Plex's published connections: all port 32400, nothing on 443

The DNS was right; the DEPENDENCY on it was the bug. And since nothing Plex
advertises listens on 443, the clients had left the Plex path entirely.

Fix: <dashed-ip>.<serverhash>.plex.direct, which is RESOLVER-INDEPENDENT -- a
public zone returning the private address embedded in the label, so every
resolver (PiHole, the router, 1.1.1.1, a DoH browser) answers it with
192.168.2.24. No split-horizon entry, no Cloudflare record and no certificate
work needed: the wildcard cert Plex already serves on 32400 covers that name.
Re-verified today with full validation (no curl -k): 200, TLS verified.

The nicer hostname cannot be made correct for clients that bypass PiHole -- a
public record resolves to the WAN address and there is no NAT hairpin, so it is
right only for PiHole users and wrong for the rest. There is no configuration of
it that is right for both. PiHole's plex-vip answer is KEPT; it is just not what
Plex advertises.

Cost restated, not buried: the value embeds the VIP and the cert hash as a
literal and nothing checks they still agree. Accepted because an unresolvable
name is a worse failure -- silent, LAN-wide, and it points users at the router.

UNVERIFIED: unreconciled, and the client-side resolver path is INFERRED -- my
public-resolver probe was broken (NXDOMAIN even for a name that resolves) so I
discarded it. The chain is the only explanation consistent with the
measurements, but the NXDOMAIN-hijack step was not observed directly.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-10-05 20:58:55 -05:00
1 parent bc7095677b
commit 874c6703f0
3 files changed
+152 -65
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

catalog/plex.yaml
+46 -64
View File
@@ -72,77 +72,59 @@ env:
secretKeyRef:
name: plex-claim
key: PLEX_CLAIM
# ADVERTISE_IP ("Custom server access URLs") — the ONE hostname this server publishes to plex.tv,
# serving BOTH local and remote clients by split-horizon DNS (operator design, 2026-10-04):
# LAN -> PiHole answers plex-vip.${BASE_DOMAIN} with ${PLEX_L4_VIP} (the tcp/32400 LoadBalancer
# in apps/wave-0/plex-remote.yaml). See catalog/pihole.yaml's split-horizon line, which
# needs a per-host override because the zone wildcard otherwise points at the GATEWAY
# VIP — measured 2026-10-04: plex-vip.${BASE_DOMAIN} resolved to the Gateway, where
# nothing listens on 32400.
# PUBLIC -> Cloudflare answers with the WAN address, NATed to the same VIP (off-repo).
# ADVERTISE_IP ("Custom server access URLs") — the connection Plex publishes to plex.tv.
#
# WHY A HOSTNAME AND NOT THE `plex.direct` FORM THIS BRIEFLY USED. The first fix for the
# 2026-10-04 outage advertised `https://<dashed-ip>.<serverhash>.plex.direct:32400`, which works
# (and is Plex's own mechanism), but it hard-codes BOTH the VIP and the server's certificate hash
# into a cluster-var — two values nothing could keep in sync, with no alert if either drifted. A
# hostname in our own zone removes that whole class of problem: ${BASE_DOMAIN} is already a knob,
# the VIP lives only in PiHole's answer, and `PLEX_ADVERTISE_URL` is deleted outright.
# THE VALUE MUST BE RESOLVABLE BY EVERY CLIENT, NOT JUST BY CLIENTS THAT USE OUR DNS. That single
# property is what this field keeps getting wrong, so it is the first thing to check before
# changing it. A Plex client resolves the advertised host ITSELF, with whatever resolver it
# happens to use — and on a real LAN that is frequently NOT PiHole: browsers ship DoH on by
# default, phones have "private DNS", guest and IoT devices often take the router's DNS, and some
# clients hard-code a public resolver.
#
# THIS IS `https`, WHICH ONLY WORKS BECAUSE WE HAND PLEX OUR OWN CERTIFICATE. Plex serves 32400
# with its `*.<serverhash>.plex.direct` cert, which does NOT cover this name — advertising
# `https://plex-vip...` against the stock cert reproduces the exact failure that caused the
# outage (measured: a storm of `CERT: incomplete TLS handshake ... certificate unknown` from real
# clients, and `curl` without `-k` failing to connect). The `lifecycle.postStart` hook below
# installs the Let's Encrypt cert for this name and points Plex at it, so the advertised URL and
# the served certificate agree. Do not change this to a bare IP, and do not drop the hook and
# leave the `https` here.
# THAT IS NOT THEORETICAL — IT TOOK PLEX DOWN FOR THE WHOLE LAN ON 2026-10-05. This field briefly
# advertised `plex-vip.${BASE_DOMAIN}`, a name that exists ONLY in PiHole's split-horizon answers
# (catalog/pihole.yaml) and in no public zone. Clients that did not use PiHole could not resolve
# it; the SOHO router answered the failed lookup with ITSELF, and every one of them landed on the
# router's own web UI and reported a certificate error for `CN = %D`. Measured 2026-10-05: the
# router serves that self-signed cert (`C = ZZ, O = %D342e7e6b, CN = %D`) on BOTH `192.168.2.1:443`
# and the WAN address, while PiHole itself was answering perfectly — both replicas returned
# `plex-vip -> ${PLEX_L4_VIP}` and `plex -> ${GATEWAY_VIP}`. The DNS was right; the DEPENDENCY on
# it was the bug. Nothing Plex advertises listens on 443, which is how we know the clients had left
# the Plex path entirely and were talking to the router.
#
# (A SAN match is not what makes Plex *work* — `httpsRequired=0` and `secureConnections` default
# to Preferred, so clients can fall back to cleartext. It is what keeps the remote leg ENCRYPTED
# instead of silently downgrading traffic that crosses the internet.)
# SO: `<dashed-ip>.<serverhash>.plex.direct`, WHICH IS RESOLVER-INDEPENDENT. This is Plex's own
# mechanism and the reason it exists: `plex.direct` is a PUBLIC zone that returns the private
# address embedded in the label, so EVERY resolver on earth — PiHole, the router, 1.1.1.1, a DoH
# browser — answers `192-168-2-24.<hash>.plex.direct` with `192.168.2.24`. It needs no
# split-horizon entry, no Cloudflare record, and no certificate work, because the wildcard cert
# Plex already serves on 32400 covers exactly this name. Re-verified live 2026-10-05 with FULL
# validation (no `curl -k`): resolves to 192.168.2.24 and `/identity` returns **200, TLS verified**.
#
# WHY NOT THE NICER HOSTNAME, since a readable name in our own zone is obviously more pleasant:
# it cannot be made to work for clients that bypass PiHole. A public record for it would resolve
# to the WAN address, and with no NAT hairpin on this router a LAN client sent there fails too —
# so the hostname is correct ONLY for clients using PiHole, and wrong for the rest. There is no
# configuration of that name that is right for both. `plex.direct` is right for both.
# (The PiHole `plex-vip` answer is KEPT — it is harmless, costs nothing, and is genuinely useful
# for reaching the VIP by name from a LAN shell. It is simply not what Plex advertises.)
#
# THE TWO EMBEDDED VALUES, and the honest cost of this choice. The dashed IP must track
# ${PLEX_L4_VIP} and `<serverhash>` is this server's certificate identity — read it from the live
# listener with
# openssl s_client -connect <vip>:32400 </dev/null | openssl x509 -noout -subject
# Nothing in this tree can derive either, and nothing checks that they still agree: envsubst
# cannot dash an IP, so this is a literal. That fragility is real and was the reason the hostname
# was tried in the first place — it is accepted here because an unresolvable name is a WORSE
# failure (silent, LAN-wide, and it points users at the router) than a stale literal, which at
# least fails loudly and only after someone changes the VIP.
#
# The image's /etc/cont-init.d/40-plex-first-run runs `setPref "customConnections"
# "${ADVERTISE_IP}"` (line 127, verified in the running image), OUTSIDE the
# PLEX_CLAIM/PlexOnlineToken guard, and its `/.firstRunComplete` marker is on the container
# OVERLAY fs rather than the /config PVC — so this is re-applied on EVERY pod start and git, not
# a web form, owns the value. Hand-typed custom URLs are what broke this service; they are now
# overwritten on every start.
#
# SCHEME IS `http` AND THAT IS TEMPORARY, 2026-10-04. It was `https`, which requires Plex to serve
# a certificate for this name — and the Certificate that would have supplied one is NOT APPLIED
# (see apps/wave-0/plex-cert.yaml's header). Advertising `https` without it reproduces the
# 2026-10-04 outage exactly: Plex answers 32400 with its `*.<serverhash>.plex.direct` cert, every
# TLS-validating client rejects the hostname, and plex.tv can publish nothing. `http` works TODAY
# and is MEASURED to: PiHole answers plex-vip.${BASE_DOMAIN} with ${PLEX_L4_VIP} (verified live
# against pihole-0, 2026-10-04) and Plex serves 32400 there.
#
# THE COST, STATED: the remote leg is unencrypted. That is tolerable only because remote access
# does not work yet at all (the WAN tcp/32400 forward is still absent), so today this carries LAN
# traffic only. RESTORE `https` IN THE SAME COMMIT that re-applies the Certificate — the two are
# one change, and splitting them is what broke this.
#
# HOW TO GET `https` BACK. The Certificate that would serve this name was applied on 2026-10-04
# and STALLED THE ENTIRE APP TIER; the file was DELETED rather than merely unregistered, because
# `tls-lifecycle-automated` (R-TLS-02) fails any Certificate left in the tree that no Flux path
# reconciles. Full post-mortem: decisions/2026-10-04-plex-unblock-app-tier.md. Three things must
# land TOGETHER:
# 1. EXEMPT cert-manager FROM `secret-provenance`. Its temporary private-key Secrets
# (`plex-vip-<random>`) are denied by `secrets-must-be-flux-managed`, which matches every
# Secret in a namespace labelled `homelab.keithjackson.dev/app-namespace: true` — `plex` HAS
# that label, `gateway` (where the working wildcard cert lives) does NOT, which is why this
# never bit the existing Certificate. `secretTemplate` cannot fix it: those Secrets come from
# the keymanager controller, not the Certificate's template. Narrow fix: an
# `exclude.any[].subjects` entry for ServiceAccount `cert-manager`/`cert-manager` (confirmed
# live as the controller's SA). `platform/kyverno/**` is a SENSITIVE path — non-author approval.
# 2. GIVE THE CERTIFICATE A NON-GATING HOME. `apps-wave-0` is `wait: true`, so Flux health-checks
# the Certificate OBJECT and a pending issuance stalls the whole tier
# (`timeout waiting for: [Certificate/plex/plex-vip status: 'InProgress']`). `optional: true`
# on the env refs below protects the POD, not the Kustomization — that distinction is the
# whole bug. Use a separate Kustomization with `wait: false`, the shape
# clusters/base/backup.yaml already uses for exactly this reason.
# 3. Flip this value back to `https`.
# PLEX_CLAIM/PlexOnlineToken guard, with its `/.firstRunComplete` marker on the container OVERLAY
# fs rather than the /config PVC — so this is re-applied on EVERY pod start and git, not a web
# form, owns the value.
- name: ADVERTISE_IP
value: http://plex-vip.${BASE_DOMAIN}:32400
value: https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400
# THE TLS MATERIAL FOR THE NAME ABOVE, as PEM text in env rather than a mounted volume.
# `plex-vip-tls` is issued by cert-manager from ClusterIssuer `letsencrypt-prod` for exactly
# `plex-vip.${BASE_DOMAIN}` (apps/wave-0/plex-cert.yaml, same wave => same reconcile as this