fix(plex): unblock the app tier — drop the Certificate that stalled apps-wave-0, advertise http

I broke the app tier with #48. Two faults, both mine, both measured after merge:

1. KYVERNO BLOCKS cert-manager IN APP NAMESPACES. The temporary private-key
   Secrets (plex-vip-<random>) were denied by
   secret-provenance/secrets-must-be-flux-managed, which matches every Secret in
   a namespace labelled homelab.keithjackson.dev/app-namespace=true and excludes
   only those labelled secret-managed-by=controller or carrying a Flux label.
   `plex` HAS that label; `gateway` -- where the working wildcard cert lives --
   does NOT. That asymmetry is why the existing Certificate was never affected
   and mine was, and I did not check it. secretTemplate cannot fix it: those
   Secrets come from the keymanager controller, not the Certificate template. So
   no CertificateRequest, Order or Challenge was ever created.

2. FLUX HEALTH-CHECKS THE CERTIFICATE. `optional: true` did its job -- the pod
   rolled and ran fine without the Secret -- but apps-wave-0 is wait:true, so
   Flux waits on the Certificate OBJECT:
     timeout waiting for: [Certificate/plex/plex-vip status: 'InProgress']
   apps-wave-0 never went Ready; wave-1/-2 and everything downstream of
   platform-config sat in DependencyNotReady behind it. I guarded the pod and
   missed the object. That distinction is the whole bug.

Third, independent: the advertised URL was https with no servable certificate --
Plex answers 32400 with its *.plex.direct cert, which cannot cover plex-vip --
the exact failure mode of the outage earlier the same day.

So: delete the Certificate (pruning it clears the health gate) and advertise
http://plex-vip:32400, which is MEASURED to work today -- PiHole answers the
name with the Plex VIP (verified against pihole-0) and Plex serves 32400 there.
http is explicitly temporary and only tolerable because remote access does not
work yet; the restore condition is written at the call site.

DELETED, not merely unregistered: my first attempt kept the file as "tracked but
not onboarded" and the gate rejected it -- tls-lifecycle-automated (R-TLS-02)
fails any Certificate no Flux path reconciles. Legal for a Deployment, not for a
TLS object. The re-enable recipe (Kyverno exemption + a wait:false Kustomization
+ the https flip, as ONE change) is recorded in catalog/plex.yaml.

Kept and verified working from #48: the hostname design, PiHole's split-horizon
answer, and the postStart hook -- LanNetworksBandwidth is now set live.

Gate: rc 3, all substantive stages pass (first attempt was rc 1 BLOCKED).

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

catalog/plex.yaml

@@ -107,8 +107,42 @@ env:
   # OVERLAY fs rather than the /config PVC — so this is re-applied on EVERY pod start and git, not
   # a web form, owns the value. Hand-typed custom URLs are what broke this service; they are now
   # overwritten on every start.
+  #
+  # SCHEME IS `http` AND THAT IS TEMPORARY, 2026-10-04. It was `https`, which requires Plex to serve
+  # a certificate for this name — and the Certificate that would have supplied one is NOT APPLIED
+  # (see apps/wave-0/plex-cert.yaml's header). Advertising `https` without it reproduces the
+  # 2026-10-04 outage exactly: Plex answers 32400 with its `*.<serverhash>.plex.direct` cert, every
+  # TLS-validating client rejects the hostname, and plex.tv can publish nothing. `http` works TODAY
+  # and is MEASURED to: PiHole answers plex-vip.${BASE_DOMAIN} with ${PLEX_L4_VIP} (verified live
+  # against pihole-0, 2026-10-04) and Plex serves 32400 there.
+  #
+  # THE COST, STATED: the remote leg is unencrypted. That is tolerable only because remote access
+  # does not work yet at all (the WAN tcp/32400 forward is still absent), so today this carries LAN
+  # traffic only. RESTORE `https` IN THE SAME COMMIT that re-applies the Certificate — the two are
+  # one change, and splitting them is what broke this.
+  #
+  # HOW TO GET `https` BACK. The Certificate that would serve this name was applied on 2026-10-04
+  # and STALLED THE ENTIRE APP TIER; the file was DELETED rather than merely unregistered, because
+  # `tls-lifecycle-automated` (R-TLS-02) fails any Certificate left in the tree that no Flux path
+  # reconciles. Full post-mortem: decisions/2026-10-04-plex-unblock-app-tier.md. Three things must
+  # land TOGETHER:
+  #   1. EXEMPT cert-manager FROM `secret-provenance`. Its temporary private-key Secrets
+  #      (`plex-vip-<random>`) are denied by `secrets-must-be-flux-managed`, which matches every
+  #      Secret in a namespace labelled `homelab.keithjackson.dev/app-namespace: true` — `plex` HAS
+  #      that label, `gateway` (where the working wildcard cert lives) does NOT, which is why this
+  #      never bit the existing Certificate. `secretTemplate` cannot fix it: those Secrets come from
+  #      the keymanager controller, not the Certificate's template. Narrow fix: an
+  #      `exclude.any[].subjects` entry for ServiceAccount `cert-manager`/`cert-manager` (confirmed
+  #      live as the controller's SA). `platform/kyverno/**` is a SENSITIVE path — non-author approval.
+  #   2. GIVE THE CERTIFICATE A NON-GATING HOME. `apps-wave-0` is `wait: true`, so Flux health-checks
+  #      the Certificate OBJECT and a pending issuance stalls the whole tier
+  #      (`timeout waiting for: [Certificate/plex/plex-vip status: 'InProgress']`). `optional: true`
+  #      on the env refs below protects the POD, not the Kustomization — that distinction is the
+  #      whole bug. Use a separate Kustomization with `wait: false`, the shape
+  #      clusters/base/backup.yaml already uses for exactly this reason.
+  #   3. Flip this value back to `https`.
   - name: ADVERTISE_IP
-    value: https://plex-vip.${BASE_DOMAIN}:32400
+    value: http://plex-vip.${BASE_DOMAIN}:32400
   # THE TLS MATERIAL FOR THE NAME ABOVE, as PEM text in env rather than a mounted volume.
   # `plex-vip-tls` is issued by cert-manager from ClusterIssuer `letsencrypt-prod` for exactly
   # `plex-vip.${BASE_DOMAIN}` (apps/wave-0/plex-cert.yaml, same wave => same reconcile as this

catalog/render.py

@@ -2209,23 +2209,6 @@ EXTRA_UNITS: dict = {
             "plex-remote.yaml",
         ],
     },
-    # Plex's own TLS certificate for the single hostname it advertises to plex.tv
-    # (apps/wave-0/plex-cert.yaml). `above: False` co-locates it in plex's OWN wave, which is the
-    # only placement available: the Certificate must land in the `plex` namespace, and that
-    # namespace is created by the rendered plex.yaml in this very wave, so it CANNOT be issued
-    # earlier. The workload tolerates the resulting gap by marking the env refs `optional` rather
-    # than by ordering -- see the header of plex-cert.yaml.
-    #
-    # Non-catalog because the ServiceDefinition schema has no field for a cert-manager Certificate
-    # (additionalProperties is false); `ingressRoutes` inherits the Gateway wildcard instead, which
-    # is the opposite case to a workload terminating its own TLS on a raw L4 port.
-    "plex-cert": {
-        "anchor": "plex",
-        "above": False,
-        "files": [
-            "plex-cert.yaml",
-        ],
-    },
 }