fix(plex): advertise a resolver-independent plex.direct name again (LAN-wide outage)

I took Plex down for the whole LAN. ADVERTISE_IP was
http://plex-vip.${BASE_DOMAIN}:32400 -- a name that exists ONLY in PiHole's
split-horizon answers and in no public zone.

A Plex client resolves the advertised host ITSELF, with whatever resolver it
uses, and on a real LAN that is frequently not PiHole: browsers ship DoH on by
default, phones have "private DNS", guest/IoT devices take the router's DNS.
Those clients could not resolve the name, the router answered the failed lookup
with ITSELF, and they all landed on its web UI -- hence the %D certificate.

Measured 2026-10-05:
  192.168.2.1:443      -> C=ZZ, O=%D342e7e6b, CN=%D   (router, LAN side)
  136.51.115.252:443   -> the same %D cert            (router, WAN side)
  pihole-0 AND pihole-1 -> plex -> .20, plex-vip -> .24   (both CORRECT)
  gateway .20:443      -> CN=*.home.keithjackson.dev, valid
  http://192.168.2.24:32400/identity -> 200
  Plex's published connections: all port 32400, nothing on 443

The DNS was right; the DEPENDENCY on it was the bug. And since nothing Plex
advertises listens on 443, the clients had left the Plex path entirely.

Fix: <dashed-ip>.<serverhash>.plex.direct, which is RESOLVER-INDEPENDENT -- a
public zone returning the private address embedded in the label, so every
resolver (PiHole, the router, 1.1.1.1, a DoH browser) answers it with
192.168.2.24. No split-horizon entry, no Cloudflare record and no certificate
work needed: the wildcard cert Plex already serves on 32400 covers that name.
Re-verified today with full validation (no curl -k): 200, TLS verified.

The nicer hostname cannot be made correct for clients that bypass PiHole -- a
public record resolves to the WAN address and there is no NAT hairpin, so it is
right only for PiHole users and wrong for the rest. There is no configuration of
it that is right for both. PiHole's plex-vip answer is KEPT; it is just not what
Plex advertises.

Cost restated, not buried: the value embeds the VIP and the cert hash as a
literal and nothing checks they still agree. Accepted because an unresolvable
name is a worse failure -- silent, LAN-wide, and it points users at the router.

UNVERIFIED: unreconciled, and the client-side resolver path is INFERRED -- my
public-resolver probe was broken (NXDOMAIN even for a name that resolves) so I
discarded it. The chain is the only explanation consistent with the
measurements, but the NXDOMAIN-hijack step was not observed directly.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

apps/wave-0/kustomization.yaml

@@ -18,7 +18,6 @@ resources:
 - pihole-dns.yaml
 - pihole-oauth2-proxy-credentials.sops.yaml
 - pihole.yaml
-- plex-cert.yaml
 - plex-claim-secret.sops.yaml
 - plex-remote.yaml
 - plex.yaml

apps/wave-0/plex-cert.yaml

@@ -1,66 +0,0 @@
----
-# =================================================================================================
-# PLEX TLS — a Let's Encrypt certificate for the ONE hostname Plex advertises to plex.tv.
-# =================================================================================================
-# HAND-WRITTEN, NOT RENDERED. A non-catalog app-tier unit anchored to the `plex` ServiceDefinition
-# and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS table — the same
-# mechanism gitea-actions, pihole-dns and plex-remote use. It is hand-written for the same reason:
-# `schemas/servicedefinition.schema.json` is `additionalProperties: false` and has NO field for a
-# cert-manager Certificate. (`ingressRoutes` gets TLS for free from the Gateway's wildcard; this is
-# the opposite case — a certificate for a workload that terminates its OWN TLS on a raw L4 port.)
-#
-# WHY PLEX NEEDS ITS OWN CERTIFICATE AT ALL. Plex serves tcp/32400 directly (not through the L7
-# Gateway — the operator chose the native path 2026-10-03, apps/wave-0/plex-remote.yaml) and it
-# presents its own `*.<serverhash>.plex.direct` certificate there. That name cannot cover
-# `plex-vip.${BASE_DOMAIN}`, the hostname we advertise, so without this the advertised `https://`
-# URL fails hostname verification for every client. That is not hypothetical: it is the shape of
-# the 2026-10-04 outage, where Plex's log carried a continuous storm of
-# `CERT: incomplete TLS handshake ... sslv3 alert certificate unknown` from real LAN clients while
-# plex.tv could publish nothing and the server vanished from the account.
-#
-# A SAN match is not what makes Plex FUNCTION — `httpsRequired=0` and `secureConnections` default
-# to Preferred, so clients can fall back to cleartext. It is what keeps the REMOTE leg encrypted
-# rather than silently downgrading traffic that crosses the internet.
-#
-# HOW IT REACHES PLEX. catalog/plex.yaml takes `tls.crt`/`tls.key` from this Secret as env
-# (`optional: true` — see below), and its `lifecycle.postStart` hook converts the PEM into the
-# PKCS#12 bundle Plex's `customCertificatePath` requires and sets the three `customCertificate*`
-# prefs. The conversion is redone on EVERY pod start, so a cert-manager RENEWAL is picked up with
-# no extra moving part and nothing to remember.
-#
-# ORDERING, AND WHY THE ENV REFS ARE OPTIONAL. This Certificate must live in the `plex` namespace
-# for the Deployment to read its Secret, and that namespace is created by the rendered
-# apps/wave-0/plex.yaml ITSELF — so this cannot be issued in an earlier tier (platform-config runs
-# before the namespace exists) and lands in the SAME reconcile as the workload. On a cold bootstrap
-# the Secret is therefore absent for as long as a DNS01 challenge takes. With hard env refs Plex
-# would sit in CreateContainerConfigError until then, and because `apps-wave-0` is `wait: true`
-# with `apps-wave-1` dependsOn it, a failed or slow issuance would STALL THE WHOLE APP TIER. Hence
-# `optional: true` on both refs (schema 1.11.0, added for exactly this): no certificate means Plex
-# still starts and serves on its own plex.direct cert, the hook logs that TLS was not installed,
-# and the next start after issuance converges.
-#
-# ISSUER: ClusterIssuer `letsencrypt-prod` (platform/cert-manager/clusterissuers.yaml), the same
-# issuer behind the Gateway's wildcard (platform/gateway/wildcard-certificate.yaml, Ready 28d on
-# prod as measured 2026-10-04). A SINGLE name is requested rather than reusing the wildcard,
-# because the wildcard's Secret lives in the `gateway` namespace and a Secret cannot be read across
-# namespaces; requesting `*.${BASE_DOMAIN}` again here would also burn a duplicate-certificate slot
-# against Let's Encrypt's weekly limit for no benefit. A single name is the least-privilege shape:
-# this key signs for `plex-vip.${BASE_DOMAIN}` and nothing else.
-apiVersion: cert-manager.io/v1
-kind: Certificate
-metadata:
-  name: plex-vip
-  namespace: plex
-  labels:
-    app.kubernetes.io/name: plex
-spec:
-  # The name Plex ADVERTISES and the name on this certificate must stay identical. They are two
-  # halves of one decision: catalog/plex.yaml's ADVERTISE_IP, the postStart hook's
-  # customCertificateDomain, this dnsNames entry, and catalog/pihole.yaml's split-horizon answer.
-  # Nothing enforces that they agree.
-  dnsNames:
-    - plex-vip.${BASE_DOMAIN}
-  secretName: plex-vip-tls
-  issuerRef:
-    kind: ClusterIssuer
-    name: letsencrypt-prod

apps/wave-0/plex.yaml

@@ -138,7 +138,7 @@ spec:
               name: plex-claim
               key: PLEX_CLAIM
         - name: ADVERTISE_IP
-          value: https://plex-vip.${BASE_DOMAIN}:32400
+          value: https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400
         - name: PLEX_TLS_CRT
           valueFrom:
             secretKeyRef:

catalog/plex.yaml

@@ -72,43 +72,59 @@ env:
       secretKeyRef:
         name: plex-claim
         key: PLEX_CLAIM
-  # ADVERTISE_IP ("Custom server access URLs") — the ONE hostname this server publishes to plex.tv,
-  # serving BOTH local and remote clients by split-horizon DNS (operator design, 2026-10-04):
-  #   LAN    -> PiHole answers plex-vip.${BASE_DOMAIN} with ${PLEX_L4_VIP} (the tcp/32400 LoadBalancer
-  #             in apps/wave-0/plex-remote.yaml). See catalog/pihole.yaml's split-horizon line, which
-  #             needs a per-host override because the zone wildcard otherwise points at the GATEWAY
-  #             VIP — measured 2026-10-04: plex-vip.${BASE_DOMAIN} resolved to the Gateway, where
-  #             nothing listens on 32400.
-  #   PUBLIC -> Cloudflare answers with the WAN address, NATed to the same VIP (off-repo).
+  # ADVERTISE_IP ("Custom server access URLs") — the connection Plex publishes to plex.tv.
   #
-  # WHY A HOSTNAME AND NOT THE `plex.direct` FORM THIS BRIEFLY USED. The first fix for the
-  # 2026-10-04 outage advertised `https://<dashed-ip>.<serverhash>.plex.direct:32400`, which works
-  # (and is Plex's own mechanism), but it hard-codes BOTH the VIP and the server's certificate hash
-  # into a cluster-var — two values nothing could keep in sync, with no alert if either drifted. A
-  # hostname in our own zone removes that whole class of problem: ${BASE_DOMAIN} is already a knob,
-  # the VIP lives only in PiHole's answer, and `PLEX_ADVERTISE_URL` is deleted outright.
+  # THE VALUE MUST BE RESOLVABLE BY EVERY CLIENT, NOT JUST BY CLIENTS THAT USE OUR DNS. That single
+  # property is what this field keeps getting wrong, so it is the first thing to check before
+  # changing it. A Plex client resolves the advertised host ITSELF, with whatever resolver it
+  # happens to use — and on a real LAN that is frequently NOT PiHole: browsers ship DoH on by
+  # default, phones have "private DNS", guest and IoT devices often take the router's DNS, and some
+  # clients hard-code a public resolver.
   #
-  # THIS IS `https`, WHICH ONLY WORKS BECAUSE WE HAND PLEX OUR OWN CERTIFICATE. Plex serves 32400
-  # with its `*.<serverhash>.plex.direct` cert, which does NOT cover this name — advertising
-  # `https://plex-vip...` against the stock cert reproduces the exact failure that caused the
-  # outage (measured: a storm of `CERT: incomplete TLS handshake ... certificate unknown` from real
-  # clients, and `curl` without `-k` failing to connect). The `lifecycle.postStart` hook below
-  # installs the Let's Encrypt cert for this name and points Plex at it, so the advertised URL and
-  # the served certificate agree. Do not change this to a bare IP, and do not drop the hook and
-  # leave the `https` here.
+  # THAT IS NOT THEORETICAL — IT TOOK PLEX DOWN FOR THE WHOLE LAN ON 2026-10-05. This field briefly
+  # advertised `plex-vip.${BASE_DOMAIN}`, a name that exists ONLY in PiHole's split-horizon answers
+  # (catalog/pihole.yaml) and in no public zone. Clients that did not use PiHole could not resolve
+  # it; the SOHO router answered the failed lookup with ITSELF, and every one of them landed on the
+  # router's own web UI and reported a certificate error for `CN = %D`. Measured 2026-10-05: the
+  # router serves that self-signed cert (`C = ZZ, O = %D342e7e6b, CN = %D`) on BOTH `192.168.2.1:443`
+  # and the WAN address, while PiHole itself was answering perfectly — both replicas returned
+  # `plex-vip -> ${PLEX_L4_VIP}` and `plex -> ${GATEWAY_VIP}`. The DNS was right; the DEPENDENCY on
+  # it was the bug. Nothing Plex advertises listens on 443, which is how we know the clients had left
+  # the Plex path entirely and were talking to the router.
   #
-  # (A SAN match is not what makes Plex *work* — `httpsRequired=0` and `secureConnections` default
-  # to Preferred, so clients can fall back to cleartext. It is what keeps the remote leg ENCRYPTED
-  # instead of silently downgrading traffic that crosses the internet.)
+  # SO: `<dashed-ip>.<serverhash>.plex.direct`, WHICH IS RESOLVER-INDEPENDENT. This is Plex's own
+  # mechanism and the reason it exists: `plex.direct` is a PUBLIC zone that returns the private
+  # address embedded in the label, so EVERY resolver on earth — PiHole, the router, 1.1.1.1, a DoH
+  # browser — answers `192-168-2-24.<hash>.plex.direct` with `192.168.2.24`. It needs no
+  # split-horizon entry, no Cloudflare record, and no certificate work, because the wildcard cert
+  # Plex already serves on 32400 covers exactly this name. Re-verified live 2026-10-05 with FULL
+  # validation (no `curl -k`): resolves to 192.168.2.24 and `/identity` returns **200, TLS verified**.
+  #
+  # WHY NOT THE NICER HOSTNAME, since a readable name in our own zone is obviously more pleasant:
+  # it cannot be made to work for clients that bypass PiHole. A public record for it would resolve
+  # to the WAN address, and with no NAT hairpin on this router a LAN client sent there fails too —
+  # so the hostname is correct ONLY for clients using PiHole, and wrong for the rest. There is no
+  # configuration of that name that is right for both. `plex.direct` is right for both.
+  # (The PiHole `plex-vip` answer is KEPT — it is harmless, costs nothing, and is genuinely useful
+  # for reaching the VIP by name from a LAN shell. It is simply not what Plex advertises.)
+  #
+  # THE TWO EMBEDDED VALUES, and the honest cost of this choice. The dashed IP must track
+  # ${PLEX_L4_VIP} and `<serverhash>` is this server's certificate identity — read it from the live
+  # listener with
+  #     openssl s_client -connect <vip>:32400 </dev/null | openssl x509 -noout -subject
+  # Nothing in this tree can derive either, and nothing checks that they still agree: envsubst
+  # cannot dash an IP, so this is a literal. That fragility is real and was the reason the hostname
+  # was tried in the first place — it is accepted here because an unresolvable name is a WORSE
+  # failure (silent, LAN-wide, and it points users at the router) than a stale literal, which at
+  # least fails loudly and only after someone changes the VIP.
   #
   # The image's /etc/cont-init.d/40-plex-first-run runs `setPref "customConnections"
   # "${ADVERTISE_IP}"` (line 127, verified in the running image), OUTSIDE the
-  # PLEX_CLAIM/PlexOnlineToken guard, and its `/.firstRunComplete` marker is on the container
-  # OVERLAY fs rather than the /config PVC — so this is re-applied on EVERY pod start and git, not
-  # a web form, owns the value. Hand-typed custom URLs are what broke this service; they are now
-  # overwritten on every start.
+  # PLEX_CLAIM/PlexOnlineToken guard, with its `/.firstRunComplete` marker on the container OVERLAY
+  # fs rather than the /config PVC — so this is re-applied on EVERY pod start and git, not a web
+  # form, owns the value.
   - name: ADVERTISE_IP
-    value: https://plex-vip.${BASE_DOMAIN}:32400
+    value: https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400
   # THE TLS MATERIAL FOR THE NAME ABOVE, as PEM text in env rather than a mounted volume.
   # `plex-vip-tls` is issued by cert-manager from ClusterIssuer `letsencrypt-prod` for exactly
   # `plex-vip.${BASE_DOMAIN}` (apps/wave-0/plex-cert.yaml, same wave => same reconcile as this

catalog/render.py

@@ -2209,23 +2209,6 @@ EXTRA_UNITS: dict = {
             "plex-remote.yaml",
         ],
     },
-    # Plex's own TLS certificate for the single hostname it advertises to plex.tv
-    # (apps/wave-0/plex-cert.yaml). `above: False` co-locates it in plex's OWN wave, which is the
-    # only placement available: the Certificate must land in the `plex` namespace, and that
-    # namespace is created by the rendered plex.yaml in this very wave, so it CANNOT be issued
-    # earlier. The workload tolerates the resulting gap by marking the env refs `optional` rather
-    # than by ordering -- see the header of plex-cert.yaml.
-    #
-    # Non-catalog because the ServiceDefinition schema has no field for a cert-manager Certificate
-    # (additionalProperties is false); `ingressRoutes` inherits the Gateway wildcard instead, which
-    # is the opposite case to a workload terminating its own TLS on a raw L4 port.
-    "plex-cert": {
-        "anchor": "plex",
-        "above": False,
-        "files": [
-            "plex-cert.yaml",
-        ],
-    },
 }
 
 

decisions/2026-10-04-plex-unblock-app-tier.md

@@ -0,0 +1,126 @@
+---
+date: 2026-10-04
+branch: fix/plex-unblock-app-tier
+title: Unblock the app tier — my Certificate stalled apps-wave-0, and the advertised https URL had no servable certificate
+---
+
+## What changed
+
+**I broke the app tier with PR #48 and this reverts the part that did it.** Two faults, both mine,
+both measured after the merge:
+
+1. **Kyverno blocks cert-manager in app namespaces.** The temporary private-key Secrets
+   (`plex-vip-<random>`) were denied by `secret-provenance/secrets-must-be-flux-managed`:
+
+       resource Secret/plex/plex-vip-rp4qj was blocked due to the following policies
+       secret-provenance:
+         secrets-must-be-flux-managed: Secrets in application namespaces must be delivered by
+         Flux from SOPS-encrypted git ... or be exempt controller-generated Secrets.
+
+   The policy matches every Secret in a namespace labelled
+   `homelab.keithjackson.dev/app-namespace: true` and excludes only Secrets labelled
+   `homelab.keithjackson.dev/secret-managed-by: controller` or carrying a Flux label. **The `plex`
+   namespace has that label; the `gateway` namespace — where the working wildcard cert lives — does
+   NOT.** That asymmetry is why the existing Certificate was never affected and mine was, and I did
+   not check it. cert-manager's temp key Secrets carry neither exempting label, and `secretTemplate`
+   cannot label them (the keymanager controller creates them, not the Certificate template). Result:
+   **no CertificateRequest, Order or Challenge was ever created.**
+
+2. **Flux health-checks the Certificate — guarding the pod was not enough.** `optional: true` did
+   exactly what it was designed to do: Plex rolled and ran fine without the Secret (verified). But
+   `apps-wave-0` is `wait: true`, so Flux waits on every applied object with a known health type:
+
+       health check failed after 5m0.045564329s: timeout waiting for:
+       [Certificate/plex/plex-vip status: 'InProgress']
+
+   apps-wave-0 never went Ready, and apps-wave-1/-2 plus everything downstream of platform-config
+   sat in `DependencyNotReady` behind it. I reasoned about the pod and missed the object.
+
+A third, independent problem the same PR introduced: **the advertised URL was `https://` with no
+servable certificate.** Plex answers 32400 with its `*.<serverhash>.plex.direct` cert, which cannot
+cover `plex-vip.${BASE_DOMAIN}` — the precise failure mode of the outage earlier that day.
+
+- **`apps/wave-0/plex-cert.yaml` DELETED**, and `plex-cert.yaml` removed from
+  `apps/wave-0/kustomization.yaml` `resources`, so Flux prunes the stuck Certificate and the health
+  gate clears. **Deleted rather than merely unregistered** — my first attempt kept the file as
+  "tracked but not onboarded" (the shape `keycloak.yaml` and `vm-demo.yaml` use) and **the repo's own
+  gate rejected it**: `tls-lifecycle-automated` (R-TLS-02) fails any Certificate in the tree that no
+  Flux path reconciles — *"nothing reconciles it, so a deleted Certificate/issuer/controller would
+  stay deleted until a human re-applied it."* That pattern is legal for a Deployment and not for a
+  TLS object, which is a distinction I had not known.
+- **`catalog/render.py`:** the now-dangling `plex-cert` EXTRA_UNITS entry removed.
+- **The re-enable recipe is recorded at the call site** — in `catalog/plex.yaml` beside the scheme
+  it is coupled to — rather than in the deleted file: the Kyverno exemption, the non-gating
+  Kustomization, and the `https` flip, as one change.
+- **`catalog/plex.yaml`:** `ADVERTISE_IP` → **`http://`**`plex-vip.${BASE_DOMAIN}:32400`, explicitly
+  temporary, with the restore condition written at the call site.
+
+**Everything else from PR #48 is kept and working** — the hostname design, the PiHole split-horizon
+answer, the `LanNetworksBandwidth` hook, the `optional` schema field, and the TLS env (inert without
+the Secret, so re-enabling is additive).
+
+## What I verified
+
+Live prod, 2026-10-04, read-only, after the #48/#50 merges.
+
+- **The stall and its exact cause**: the health-check timeout message above, and the full dependency
+  cascade (`cilium` → "dependency 'flux-system/flux-system' revision is not up to date", everything
+  else `DependencyNotReady`). The root `cilium` wait was a normal revision-gated cascade; the
+  *Certificate* was the real blocker.
+- **The Kyverno denial text, verbatim**, from the cert-manager controller log, repeating every ~30s
+  across at least 15 distinct Secret names.
+- **The policy's own structure**, read from `platform/kyverno/admission/secret-provenance.yaml`:
+  match = Secrets in `app-namespace: true` namespaces; exclude = `secret-managed-by: controller` OR
+  a `kustomize.toolkit.fluxcd.io/name` label; validate = unconditional `deny`.
+- **The namespace-label asymmetry**: `plex` carries
+  `homelab.keithjackson.dev/app-namespace: "true"`; `gateway` does not.
+- **cert-manager's controller ServiceAccount is `cert-manager`** in namespace `cert-manager` — the
+  identity a future exemption should name.
+- **`optional: true` worked**: the new pod (`plex-c4f87c84d-zgjq5`) is `1/1 Running` with no Secret
+  present, and `/config/certs` was never created.
+- **The postStart hook RAN and took effect** — the thing the whole hook was written for:
+  `LanNetworksBandwidth="10.0.0.0/8,192.168.0.0/16"` is now set in the live `Preferences.xml`, and
+  `customConnections="https://plex-vip.home.keithjackson.dev:32400"` was applied from the env. The
+  certificate prefs are correctly **ABSENT**, i.e. the hook's "no bundle → leave them alone" guard
+  behaved as designed.
+- **PiHole's split-horizon answer is correct**, asked of `pihole-0` directly:
+  `plex-vip.home.keithjackson.dev -> 192.168.2.24`, alongside `plex -> .20` and `crafty-mc -> .22`,
+  and the live `/etc/pihole/dnsmasq.conf` carries
+  `address=/plex-vip.home.keithjackson.dev/192.168.2.24`. **This corrects my own earlier check**,
+  which read `.20` and which I nearly reported as a failure — that test resolved via CLUSTER DNS
+  (CoreDNS) from inside a pod and never consulted PiHole at all.
+- **The gate caught my first attempt and I fixed it rather than working around it.** Keeping the
+  file unregistered produced `MERGE GATE: BLOCKED` (rc 1) with
+  `CHECK FAIL (tls-lifecycle-automated): 1 of 612 assertions failed`. After deleting the file and
+  the dangling EXTRA_UNITS entry: **rc 3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`,
+  `render-golden`, `l1` and `approval-policy` all **pass** — the rc 3 being only the two known
+  environmental skips (`gitea-settings`, `conformance` L2/L3). **Not green. Reported as rc 3.**
+
+## What I did NOT verify
+
+- **Whether this actually unblocks the tier.** Unreconciled at time of writing. The reasoning is
+  that pruning the Certificate removes the only failing health check, but I have not seen
+  apps-wave-0 go Ready.
+- **Whether `http://` restores client connectivity.** PiHole resolves the name and Plex serves 32400
+  (both measured), but I have not watched a Plex client complete a connection over the advertised
+  URL, and I did not re-check whether plex.tv publishes it.
+- **THE REMOTE BADGE — the symptom that started this.** `LanNetworksBandwidth` is set, but the log
+  still tags traffic `Allowed Network (WAN)`: **82 WAN, 0 LAN**, with entries continuing after the
+  pref was written. So either Plex needs a restart for it to take effect, or that log tag is not
+  driven by this pref at all. **I do not know which, and the user-visible badge is the only real
+  test** — it is in the Plex client, which I cannot see.
+- **The Hubble flow export is NOT active.** The cilium agents are still the pre-merge pods, so the
+  `filePath: stdout` question is still open and no flow has been exported. The Cilium rollout — and
+  its LAN-DNS risk, since prod-3 holds the Gateway, PiHole-DNS and Plex VIPs — is still AHEAD.
+- **Whether a cert-manager exemption is the right shape**, or whether it widens
+  `secret-provenance` more than the operator wants. It is `platform/kyverno/**`, a sensitive path,
+  and deliberately not attempted here.
+- **Whether the pruned Certificate leaves residue** — I did not check for orphaned
+  `plex-vip-<random>` Secrets or CertificateRequests after the prune. Note Kyverno DENIED every one
+  of those Secrets, so there may be nothing to clean, but that is an inference, not a check.
+- **Whether `tls-lifecycle-automated` would also have objected to the Certificate living in a
+  `wait: false` Kustomization** (the recommended re-enable shape). It demands a reconciling Flux
+  path, which that provides, but I did not test it.
+- **The unencrypted advertised URL.** `http://` is tolerable only while remote access does not work
+  (no WAN forward). Nothing enforces that assumption, and nothing will complain if `http` is still
+  there the day the port-forward appears.

decisions/2026-10-05-plex-resolver-independent-advertise.md

@@ -0,0 +1,105 @@
+---
+date: 2026-10-05
+branch: fix/plex-resolver-independent-advertise
+title: Plex down LAN-wide — the advertised name existed only in PiHole, so clients that bypass it landed on the router's %D cert
+---
+
+## What changed
+
+**I took Plex down for the whole LAN.** Reported symptom: every internal client showed a certificate
+error with `%D` as the subject name and could not connect.
+
+**Root cause: the advertised connection depended on our own resolver.** `ADVERTISE_IP` was
+`http://plex-vip.${BASE_DOMAIN}:32400` — a name that exists ONLY in PiHole's split-horizon answers
+and in no public zone. A Plex client resolves the advertised host *itself*, with whatever resolver it
+happens to use, and on a real LAN that is frequently not PiHole: browsers ship DoH on by default,
+phones have "private DNS", guest/IoT devices take the router's DNS. Those clients could not resolve
+the name, the SOHO router answered the failed lookup with **itself**, and they all landed on the
+router's web UI.
+
+| Measured 2026-10-05 | Result |
+|---|---|
+| `192.168.2.1:443` (router, LAN side) | `C = ZZ, ST = Somewhere, O = %D342e7e6b, CN = %D` |
+| `136.51.115.252:443` (router, WAN side) | the same `%D` cert |
+| PiHole **pihole-0** | `plex -> 192.168.2.20`, `plex-vip -> 192.168.2.24` ✅ |
+| PiHole **pihole-1** | identical ✅ (both behind the `.23` VIP) |
+| Gateway `.20:443` cert | `CN = *.home.keithjackson.dev`, valid to 2026-12-12 ✅ |
+| `http://192.168.2.24:32400/identity` | `200` ✅ |
+| Plex's published connections | all port **32400** — nothing on 443 |
+
+**The DNS was right; the dependency on it was the bug.** PiHole answered correctly from both
+replicas the entire time. And because nothing Plex advertises listens on 443, the clients had plainly
+left the Plex path altogether and were talking to the router.
+
+**The fix: go back to `<dashed-ip>.<serverhash>.plex.direct`, which is RESOLVER-INDEPENDENT.** This
+is Plex's own mechanism and the reason it exists — `plex.direct` is a public zone that returns the
+private address embedded in the label, so *every* resolver (PiHole, the router, 1.1.1.1, a DoH
+browser) answers `192-168-2-24.<hash>.plex.direct` with `192.168.2.24`. It needs no split-horizon
+entry, no Cloudflare record and no certificate work, because the wildcard cert Plex already serves on
+32400 covers exactly that name.
+
+- **`catalog/plex.yaml`:** `ADVERTISE_IP` →
+  `https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400`, with the
+  resolver-independence requirement written at the top of the block as the first thing to check
+  before anyone changes this field again.
+- **PiHole's `plex-vip` answer is KEPT.** It is harmless, costs nothing, and is useful for reaching
+  the VIP by name from a LAN shell. It is simply not what Plex advertises.
+
+**Why not keep the nicer hostname.** It cannot be made correct for clients that bypass PiHole: a
+public record for it resolves to the WAN address, and with no NAT hairpin on this router a LAN client
+sent there fails too. The name is right only for PiHole users and wrong for everyone else, and there
+is no configuration of it that is right for both. `plex.direct` is right for both.
+
+**The honest cost, restated rather than buried:** the value embeds both the VIP and the server's
+certificate hash as a literal — envsubst cannot dash an IP and nothing in the tree can derive the
+hash, so nothing checks that they still agree. That fragility is exactly why the hostname was tried.
+It is accepted because an unresolvable name is a *worse* failure: silent, LAN-wide, and it points
+users at their router.
+
+## What I verified
+
+Live prod, 2026-10-05, read-only.
+
+- **Every row of the table above**, including the router's `%D` cert on both its LAN and WAN sides —
+  which is what identifies the router as what the clients were reaching.
+- **THE FIX ITSELF, with full TLS validation (no `curl -k`)**: the plex.direct name resolves to
+  `192.168.2.24` and `/identity` returns **200, TLS verified**. Re-checked today, not carried over
+  from yesterday's measurement.
+- **Both PiHole replicas are identical and correct**, with the same five `address=` lines live in
+  `/etc/pihole/dnsmasq.conf`, and both IPs present behind the `pihole-dns-tcp`/`-udp` endpoints. I
+  checked only pihole-0 at first and deliberately went back for pihole-1, because a split-brain pair
+  would have produced exactly this intermittent-looking symptom.
+- **The cluster was never the problem:** all eight LoadBalancer VIPs still assigned, L2
+  announcements intact (prod-3 holds `.20`/`.23`/`.24`), `wildcard-home` Certificate `Ready=True`,
+  and PR #51 had reconciled (`customConnections="http://plex-vip.home.keithjackson.dev:32400"` was
+  live, confirming the bad value was in effect).
+- **`LanNetworksBandwidth="10.0.0.0/8,192.168.0.0/16"` is still set**, so the postStart hook survived
+  the roll and keeps re-asserting.
+- **The Cilium rollout from #50 STILL has not happened** — agents are unchanged since
+  2026-10-04T07:11, so the Hubble export is inactive and `filePath: stdout` remains unproven.
+
+## What I did NOT verify
+
+- **THE OUTCOME. Unreconciled.** No client has been observed connecting after this change. The
+  argument is that the advertised name now resolves for every resolver and serves a matching
+  certificate — both measured — but I have not seen a Plex client complete a connection, and I
+  cannot see the user's clients at all.
+- **THE CLIENT-SIDE RESOLVER PATH IS INFERRED, NOT OBSERVED.** I could not query a public resolver
+  from here — my attempt returned NXDOMAIN even for a `plex.direct` name that certainly resolves, so
+  the probe was broken (egress policy or a missing tool) and I discarded it rather than report it.
+  The chain "client bypasses PiHole → NXDOMAIN → router answers with itself → `%D`" is the only
+  explanation consistent with all the measurements, but the NXDOMAIN-hijack step in particular is
+  deduced from the router serving `%D` on its LAN address, not from watching a lookup.
+- **Which clients bypass PiHole, or how many.** If in fact every client uses PiHole, then my
+  explanation is wrong and something else sent them to the router — in which case the next thing to
+  check is whether the clients were using `plex.home.keithjackson.dev` (port 443, the Gateway) rather
+  than an advertised connection at all.
+- **Whether plex.tv will re-publish promptly** after the value changes, and whether stale connection
+  lists are cached on clients. A client holding the old list may keep failing until it refreshes.
+- **Mixed content.** The previous `http://` value would also have been blocked by the browser for
+  anyone using `https://app.plex.tv`; that is fixed incidentally by returning to `https`, but I did
+  not test it.
+- **The Remote badge — the symptom that began this whole sequence — remains unconfirmed.**
+  `LanNetworksBandwidth` is set, but Plex still tags traffic `Allowed Network (WAN)`. Unchanged and
+  still unexplained.
+- **`ManualPortMappingMode="1"` with an absent port** is still there, untouched.