Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
  jakman85andClaude Opus 5
|
874c6703f0 | ||
|
jakman85
|
bc7095677b |
No files matched your search
@@ -138,7 +138,7 @@ spec:
|
||||
name: plex-claim
|
||||
key: PLEX_CLAIM
|
||||
- name: ADVERTISE_IP
|
||||
value: http://plex-vip.${BASE_DOMAIN}:32400
|
||||
value: https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400
|
||||
- name: PLEX_TLS_CRT
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
|
||||
+46
-64
@@ -72,77 +72,59 @@ env:
|
||||
secretKeyRef:
|
||||
name: plex-claim
|
||||
key: PLEX_CLAIM
|
||||
# ADVERTISE_IP ("Custom server access URLs") — the ONE hostname this server publishes to plex.tv,
|
||||
# serving BOTH local and remote clients by split-horizon DNS (operator design, 2026-10-04):
|
||||
# LAN -> PiHole answers plex-vip.${BASE_DOMAIN} with ${PLEX_L4_VIP} (the tcp/32400 LoadBalancer
|
||||
# in apps/wave-0/plex-remote.yaml). See catalog/pihole.yaml's split-horizon line, which
|
||||
# needs a per-host override because the zone wildcard otherwise points at the GATEWAY
|
||||
# VIP — measured 2026-10-04: plex-vip.${BASE_DOMAIN} resolved to the Gateway, where
|
||||
# nothing listens on 32400.
|
||||
# PUBLIC -> Cloudflare answers with the WAN address, NATed to the same VIP (off-repo).
|
||||
# ADVERTISE_IP ("Custom server access URLs") — the connection Plex publishes to plex.tv.
|
||||
#
|
||||
# WHY A HOSTNAME AND NOT THE `plex.direct` FORM THIS BRIEFLY USED. The first fix for the
|
||||
# 2026-10-04 outage advertised `https://<dashed-ip>.<serverhash>.plex.direct:32400`, which works
|
||||
# (and is Plex's own mechanism), but it hard-codes BOTH the VIP and the server's certificate hash
|
||||
# into a cluster-var — two values nothing could keep in sync, with no alert if either drifted. A
|
||||
# hostname in our own zone removes that whole class of problem: ${BASE_DOMAIN} is already a knob,
|
||||
# the VIP lives only in PiHole's answer, and `PLEX_ADVERTISE_URL` is deleted outright.
|
||||
# THE VALUE MUST BE RESOLVABLE BY EVERY CLIENT, NOT JUST BY CLIENTS THAT USE OUR DNS. That single
|
||||
# property is what this field keeps getting wrong, so it is the first thing to check before
|
||||
# changing it. A Plex client resolves the advertised host ITSELF, with whatever resolver it
|
||||
# happens to use — and on a real LAN that is frequently NOT PiHole: browsers ship DoH on by
|
||||
# default, phones have "private DNS", guest and IoT devices often take the router's DNS, and some
|
||||
# clients hard-code a public resolver.
|
||||
#
|
||||
# THIS IS `https`, WHICH ONLY WORKS BECAUSE WE HAND PLEX OUR OWN CERTIFICATE. Plex serves 32400
|
||||
# with its `*.<serverhash>.plex.direct` cert, which does NOT cover this name — advertising
|
||||
# `https://plex-vip...` against the stock cert reproduces the exact failure that caused the
|
||||
# outage (measured: a storm of `CERT: incomplete TLS handshake ... certificate unknown` from real
|
||||
# clients, and `curl` without `-k` failing to connect). The `lifecycle.postStart` hook below
|
||||
# installs the Let's Encrypt cert for this name and points Plex at it, so the advertised URL and
|
||||
# the served certificate agree. Do not change this to a bare IP, and do not drop the hook and
|
||||
# leave the `https` here.
|
||||
# THAT IS NOT THEORETICAL — IT TOOK PLEX DOWN FOR THE WHOLE LAN ON 2026-10-05. This field briefly
|
||||
# advertised `plex-vip.${BASE_DOMAIN}`, a name that exists ONLY in PiHole's split-horizon answers
|
||||
# (catalog/pihole.yaml) and in no public zone. Clients that did not use PiHole could not resolve
|
||||
# it; the SOHO router answered the failed lookup with ITSELF, and every one of them landed on the
|
||||
# router's own web UI and reported a certificate error for `CN = %D`. Measured 2026-10-05: the
|
||||
# router serves that self-signed cert (`C = ZZ, O = %D342e7e6b, CN = %D`) on BOTH `192.168.2.1:443`
|
||||
# and the WAN address, while PiHole itself was answering perfectly — both replicas returned
|
||||
# `plex-vip -> ${PLEX_L4_VIP}` and `plex -> ${GATEWAY_VIP}`. The DNS was right; the DEPENDENCY on
|
||||
# it was the bug. Nothing Plex advertises listens on 443, which is how we know the clients had left
|
||||
# the Plex path entirely and were talking to the router.
|
||||
#
|
||||
# (A SAN match is not what makes Plex *work* — `httpsRequired=0` and `secureConnections` default
|
||||
# to Preferred, so clients can fall back to cleartext. It is what keeps the remote leg ENCRYPTED
|
||||
# instead of silently downgrading traffic that crosses the internet.)
|
||||
# SO: `<dashed-ip>.<serverhash>.plex.direct`, WHICH IS RESOLVER-INDEPENDENT. This is Plex's own
|
||||
# mechanism and the reason it exists: `plex.direct` is a PUBLIC zone that returns the private
|
||||
# address embedded in the label, so EVERY resolver on earth — PiHole, the router, 1.1.1.1, a DoH
|
||||
# browser — answers `192-168-2-24.<hash>.plex.direct` with `192.168.2.24`. It needs no
|
||||
# split-horizon entry, no Cloudflare record, and no certificate work, because the wildcard cert
|
||||
# Plex already serves on 32400 covers exactly this name. Re-verified live 2026-10-05 with FULL
|
||||
# validation (no `curl -k`): resolves to 192.168.2.24 and `/identity` returns **200, TLS verified**.
|
||||
#
|
||||
# WHY NOT THE NICER HOSTNAME, since a readable name in our own zone is obviously more pleasant:
|
||||
# it cannot be made to work for clients that bypass PiHole. A public record for it would resolve
|
||||
# to the WAN address, and with no NAT hairpin on this router a LAN client sent there fails too —
|
||||
# so the hostname is correct ONLY for clients using PiHole, and wrong for the rest. There is no
|
||||
# configuration of that name that is right for both. `plex.direct` is right for both.
|
||||
# (The PiHole `plex-vip` answer is KEPT — it is harmless, costs nothing, and is genuinely useful
|
||||
# for reaching the VIP by name from a LAN shell. It is simply not what Plex advertises.)
|
||||
#
|
||||
# THE TWO EMBEDDED VALUES, and the honest cost of this choice. The dashed IP must track
|
||||
# ${PLEX_L4_VIP} and `<serverhash>` is this server's certificate identity — read it from the live
|
||||
# listener with
|
||||
# openssl s_client -connect <vip>:32400 </dev/null | openssl x509 -noout -subject
|
||||
# Nothing in this tree can derive either, and nothing checks that they still agree: envsubst
|
||||
# cannot dash an IP, so this is a literal. That fragility is real and was the reason the hostname
|
||||
# was tried in the first place — it is accepted here because an unresolvable name is a WORSE
|
||||
# failure (silent, LAN-wide, and it points users at the router) than a stale literal, which at
|
||||
# least fails loudly and only after someone changes the VIP.
|
||||
#
|
||||
# The image's /etc/cont-init.d/40-plex-first-run runs `setPref "customConnections"
|
||||
# "${ADVERTISE_IP}"` (line 127, verified in the running image), OUTSIDE the
|
||||
# PLEX_CLAIM/PlexOnlineToken guard, and its `/.firstRunComplete` marker is on the container
|
||||
# OVERLAY fs rather than the /config PVC — so this is re-applied on EVERY pod start and git, not
|
||||
# a web form, owns the value. Hand-typed custom URLs are what broke this service; they are now
|
||||
# overwritten on every start.
|
||||
#
|
||||
# SCHEME IS `http` AND THAT IS TEMPORARY, 2026-10-04. It was `https`, which requires Plex to serve
|
||||
# a certificate for this name — and the Certificate that would have supplied one is NOT APPLIED
|
||||
# (see apps/wave-0/plex-cert.yaml's header). Advertising `https` without it reproduces the
|
||||
# 2026-10-04 outage exactly: Plex answers 32400 with its `*.<serverhash>.plex.direct` cert, every
|
||||
# TLS-validating client rejects the hostname, and plex.tv can publish nothing. `http` works TODAY
|
||||
# and is MEASURED to: PiHole answers plex-vip.${BASE_DOMAIN} with ${PLEX_L4_VIP} (verified live
|
||||
# against pihole-0, 2026-10-04) and Plex serves 32400 there.
|
||||
#
|
||||
# THE COST, STATED: the remote leg is unencrypted. That is tolerable only because remote access
|
||||
# does not work yet at all (the WAN tcp/32400 forward is still absent), so today this carries LAN
|
||||
# traffic only. RESTORE `https` IN THE SAME COMMIT that re-applies the Certificate — the two are
|
||||
# one change, and splitting them is what broke this.
|
||||
#
|
||||
# HOW TO GET `https` BACK. The Certificate that would serve this name was applied on 2026-10-04
|
||||
# and STALLED THE ENTIRE APP TIER; the file was DELETED rather than merely unregistered, because
|
||||
# `tls-lifecycle-automated` (R-TLS-02) fails any Certificate left in the tree that no Flux path
|
||||
# reconciles. Full post-mortem: decisions/2026-10-04-plex-unblock-app-tier.md. Three things must
|
||||
# land TOGETHER:
|
||||
# 1. EXEMPT cert-manager FROM `secret-provenance`. Its temporary private-key Secrets
|
||||
# (`plex-vip-<random>`) are denied by `secrets-must-be-flux-managed`, which matches every
|
||||
# Secret in a namespace labelled `homelab.keithjackson.dev/app-namespace: true` — `plex` HAS
|
||||
# that label, `gateway` (where the working wildcard cert lives) does NOT, which is why this
|
||||
# never bit the existing Certificate. `secretTemplate` cannot fix it: those Secrets come from
|
||||
# the keymanager controller, not the Certificate's template. Narrow fix: an
|
||||
# `exclude.any[].subjects` entry for ServiceAccount `cert-manager`/`cert-manager` (confirmed
|
||||
# live as the controller's SA). `platform/kyverno/**` is a SENSITIVE path — non-author approval.
|
||||
# 2. GIVE THE CERTIFICATE A NON-GATING HOME. `apps-wave-0` is `wait: true`, so Flux health-checks
|
||||
# the Certificate OBJECT and a pending issuance stalls the whole tier
|
||||
# (`timeout waiting for: [Certificate/plex/plex-vip status: 'InProgress']`). `optional: true`
|
||||
# on the env refs below protects the POD, not the Kustomization — that distinction is the
|
||||
# whole bug. Use a separate Kustomization with `wait: false`, the shape
|
||||
# clusters/base/backup.yaml already uses for exactly this reason.
|
||||
# 3. Flip this value back to `https`.
|
||||
# PLEX_CLAIM/PlexOnlineToken guard, with its `/.firstRunComplete` marker on the container OVERLAY
|
||||
# fs rather than the /config PVC — so this is re-applied on EVERY pod start and git, not a web
|
||||
# form, owns the value.
|
||||
- name: ADVERTISE_IP
|
||||
value: http://plex-vip.${BASE_DOMAIN}:32400
|
||||
value: https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400
|
||||
# THE TLS MATERIAL FOR THE NAME ABOVE, as PEM text in env rather than a mounted volume.
|
||||
# `plex-vip-tls` is issued by cert-manager from ClusterIssuer `letsencrypt-prod` for exactly
|
||||
# `plex-vip.${BASE_DOMAIN}` (apps/wave-0/plex-cert.yaml, same wave => same reconcile as this
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
---
|
||||
date: 2026-10-05
|
||||
branch: fix/plex-resolver-independent-advertise
|
||||
title: Plex down LAN-wide — the advertised name existed only in PiHole, so clients that bypass it landed on the router's %D cert
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
**I took Plex down for the whole LAN.** Reported symptom: every internal client showed a certificate
|
||||
error with `%D` as the subject name and could not connect.
|
||||
|
||||
**Root cause: the advertised connection depended on our own resolver.** `ADVERTISE_IP` was
|
||||
`http://plex-vip.${BASE_DOMAIN}:32400` — a name that exists ONLY in PiHole's split-horizon answers
|
||||
and in no public zone. A Plex client resolves the advertised host *itself*, with whatever resolver it
|
||||
happens to use, and on a real LAN that is frequently not PiHole: browsers ship DoH on by default,
|
||||
phones have "private DNS", guest/IoT devices take the router's DNS. Those clients could not resolve
|
||||
the name, the SOHO router answered the failed lookup with **itself**, and they all landed on the
|
||||
router's web UI.
|
||||
|
||||
| Measured 2026-10-05 | Result |
|
||||
|---|---|
|
||||
| `192.168.2.1:443` (router, LAN side) | `C = ZZ, ST = Somewhere, O = %D342e7e6b, CN = %D` |
|
||||
| `136.51.115.252:443` (router, WAN side) | the same `%D` cert |
|
||||
| PiHole **pihole-0** | `plex -> 192.168.2.20`, `plex-vip -> 192.168.2.24` ✅ |
|
||||
| PiHole **pihole-1** | identical ✅ (both behind the `.23` VIP) |
|
||||
| Gateway `.20:443` cert | `CN = *.home.keithjackson.dev`, valid to 2026-12-12 ✅ |
|
||||
| `http://192.168.2.24:32400/identity` | `200` ✅ |
|
||||
| Plex's published connections | all port **32400** — nothing on 443 |
|
||||
|
||||
**The DNS was right; the dependency on it was the bug.** PiHole answered correctly from both
|
||||
replicas the entire time. And because nothing Plex advertises listens on 443, the clients had plainly
|
||||
left the Plex path altogether and were talking to the router.
|
||||
|
||||
**The fix: go back to `<dashed-ip>.<serverhash>.plex.direct`, which is RESOLVER-INDEPENDENT.** This
|
||||
is Plex's own mechanism and the reason it exists — `plex.direct` is a public zone that returns the
|
||||
private address embedded in the label, so *every* resolver (PiHole, the router, 1.1.1.1, a DoH
|
||||
browser) answers `192-168-2-24.<hash>.plex.direct` with `192.168.2.24`. It needs no split-horizon
|
||||
entry, no Cloudflare record and no certificate work, because the wildcard cert Plex already serves on
|
||||
32400 covers exactly that name.
|
||||
|
||||
- **`catalog/plex.yaml`:** `ADVERTISE_IP` →
|
||||
`https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400`, with the
|
||||
resolver-independence requirement written at the top of the block as the first thing to check
|
||||
before anyone changes this field again.
|
||||
- **PiHole's `plex-vip` answer is KEPT.** It is harmless, costs nothing, and is useful for reaching
|
||||
the VIP by name from a LAN shell. It is simply not what Plex advertises.
|
||||
|
||||
**Why not keep the nicer hostname.** It cannot be made correct for clients that bypass PiHole: a
|
||||
public record for it resolves to the WAN address, and with no NAT hairpin on this router a LAN client
|
||||
sent there fails too. The name is right only for PiHole users and wrong for everyone else, and there
|
||||
is no configuration of it that is right for both. `plex.direct` is right for both.
|
||||
|
||||
**The honest cost, restated rather than buried:** the value embeds both the VIP and the server's
|
||||
certificate hash as a literal — envsubst cannot dash an IP and nothing in the tree can derive the
|
||||
hash, so nothing checks that they still agree. That fragility is exactly why the hostname was tried.
|
||||
It is accepted because an unresolvable name is a *worse* failure: silent, LAN-wide, and it points
|
||||
users at their router.
|
||||
|
||||
## What I verified
|
||||
|
||||
Live prod, 2026-10-05, read-only.
|
||||
|
||||
- **Every row of the table above**, including the router's `%D` cert on both its LAN and WAN sides —
|
||||
which is what identifies the router as what the clients were reaching.
|
||||
- **THE FIX ITSELF, with full TLS validation (no `curl -k`)**: the plex.direct name resolves to
|
||||
`192.168.2.24` and `/identity` returns **200, TLS verified**. Re-checked today, not carried over
|
||||
from yesterday's measurement.
|
||||
- **Both PiHole replicas are identical and correct**, with the same five `address=` lines live in
|
||||
`/etc/pihole/dnsmasq.conf`, and both IPs present behind the `pihole-dns-tcp`/`-udp` endpoints. I
|
||||
checked only pihole-0 at first and deliberately went back for pihole-1, because a split-brain pair
|
||||
would have produced exactly this intermittent-looking symptom.
|
||||
- **The cluster was never the problem:** all eight LoadBalancer VIPs still assigned, L2
|
||||
announcements intact (prod-3 holds `.20`/`.23`/`.24`), `wildcard-home` Certificate `Ready=True`,
|
||||
and PR #51 had reconciled (`customConnections="http://plex-vip.home.keithjackson.dev:32400"` was
|
||||
live, confirming the bad value was in effect).
|
||||
- **`LanNetworksBandwidth="10.0.0.0/8,192.168.0.0/16"` is still set**, so the postStart hook survived
|
||||
the roll and keeps re-asserting.
|
||||
- **The Cilium rollout from #50 STILL has not happened** — agents are unchanged since
|
||||
2026-10-04T07:11, so the Hubble export is inactive and `filePath: stdout` remains unproven.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **THE OUTCOME. Unreconciled.** No client has been observed connecting after this change. The
|
||||
argument is that the advertised name now resolves for every resolver and serves a matching
|
||||
certificate — both measured — but I have not seen a Plex client complete a connection, and I
|
||||
cannot see the user's clients at all.
|
||||
- **THE CLIENT-SIDE RESOLVER PATH IS INFERRED, NOT OBSERVED.** I could not query a public resolver
|
||||
from here — my attempt returned NXDOMAIN even for a `plex.direct` name that certainly resolves, so
|
||||
the probe was broken (egress policy or a missing tool) and I discarded it rather than report it.
|
||||
The chain "client bypasses PiHole → NXDOMAIN → router answers with itself → `%D`" is the only
|
||||
explanation consistent with all the measurements, but the NXDOMAIN-hijack step in particular is
|
||||
deduced from the router serving `%D` on its LAN address, not from watching a lookup.
|
||||
- **Which clients bypass PiHole, or how many.** If in fact every client uses PiHole, then my
|
||||
explanation is wrong and something else sent them to the router — in which case the next thing to
|
||||
check is whether the clients were using `plex.home.keithjackson.dev` (port 443, the Gateway) rather
|
||||
than an advertised connection at all.
|
||||
- **Whether plex.tv will re-publish promptly** after the value changes, and whether stale connection
|
||||
lists are cached on clients. A client holding the old list may keep failing until it refreshes.
|
||||
- **Mixed content.** The previous `http://` value would also have been blocked by the browser for
|
||||
anyone using `https://app.plex.tv`; that is fixed incidentally by returning to `https`, but I did
|
||||
not test it.
|
||||
- **The Remote badge — the symptom that began this whole sequence — remains unconfirmed.**
|
||||
`LanNetworksBandwidth` is set, but Plex still tags traffic `Allowed Network (WAN)`. Unchanged and
|
||||
still unexplained.
|
||||
- **`ManualPortMappingMode="1"` with an absent port** is still there, untouched.
|
||||
Reference in new issue
Block a user