fix(plex): unblock the app tier — drop the Certificate that stalled apps-wave-0, advertise http

I broke the app tier with #48. Two faults, both mine, both measured after merge: 1. KYVERNO BLOCKS cert-manager IN APP NAMESPACES. The temporary private-key Secrets (plex-vip-<random>) were denied by secret-provenance/secrets-must-be-flux-managed, which matches every Secret in a namespace labelled homelab.keithjackson.dev/app-namespace=true and excludes only those labelled secret-managed-by=controller or carrying a Flux label. `plex` HAS that label; `gateway` -- where the working wildcard cert lives -- does NOT. That asymmetry is why the existing Certificate was never affected and mine was, and I did not check it. secretTemplate cannot fix it: those Secrets come from the keymanager controller, not the Certificate template. So no CertificateRequest, Order or Challenge was ever created. 2. FLUX HEALTH-CHECKS THE CERTIFICATE. `optional: true` did its job -- the pod rolled and ran fine without the Secret -- but apps-wave-0 is wait:true, so Flux waits on the Certificate OBJECT: timeout waiting for: [Certificate/plex/plex-vip status: 'InProgress'] apps-wave-0 never went Ready; wave-1/-2 and everything downstream of platform-config sat in DependencyNotReady behind it. I guarded the pod and missed the object. That distinction is the whole bug. Third, independent: the advertised URL was https with no servable certificate -- Plex answers 32400 with its *.plex.direct cert, which cannot cover plex-vip -- the exact failure mode of the outage earlier the same day. So: delete the Certificate (pruning it clears the health gate) and advertise http://plex-vip:32400, which is MEASURED to work today -- PiHole answers the name with the Plex VIP (verified against pihole-0) and Plex serves 32400 there. http is explicitly temporary and only tolerable because remote access does not work yet; the restore condition is written at the call site. DELETED, not merely unregistered: my first attempt kept the file as "tracked but not onboarded" and the gate rejected it -- tls-lifecycle-automated (R-TLS-02) fails any Certificate no Flux path reconciles. Legal for a Deployment, not for a TLS object. The re-enable recipe (Kyverno exemption + a wait:false Kustomization + the https flip, as ONE change) is recorded in catalog/plex.yaml. Kept and verified working from #48: the hostname design, PiHole's split-horizon answer, and the postStart hook -- LanNetworksBandwidth is now set live. Gate: rc 3, all substantive stages pass (first attempt was rc 1 BLOCKED). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-04 17:10:25 -05:00
1 parent c163deaef9
commit a0e6ef145f
@@ -18,7 +18,6 @@ resources:
 - pihole-dns.yaml
 - pihole-oauth2-proxy-credentials.sops.yaml
 - pihole.yaml
- plex-cert.yaml
 - plex-claim-secret.sops.yaml
 - plex-remote.yaml
 - plex.yaml
@@ -1,66 +0,0 @@
---
-# =================================================================================================
-# PLEX TLS — a Let's Encrypt certificate for the ONE hostname Plex advertises to plex.tv.
-# =================================================================================================
-# HAND-WRITTEN, NOT RENDERED. A non-catalog app-tier unit anchored to the `plex` ServiceDefinition
-# and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS table — the same
-# mechanism gitea-actions, pihole-dns and plex-remote use. It is hand-written for the same reason:
-# `schemas/servicedefinition.schema.json` is `additionalProperties: false` and has NO field for a
-# cert-manager Certificate. (`ingressRoutes` gets TLS for free from the Gateway's wildcard; this is
-# the opposite case — a certificate for a workload that terminates its OWN TLS on a raw L4 port.)
-#
-# WHY PLEX NEEDS ITS OWN CERTIFICATE AT ALL. Plex serves tcp/32400 directly (not through the L7
-# Gateway — the operator chose the native path 2026-10-03, apps/wave-0/plex-remote.yaml) and it
-# presents its own `*.<serverhash>.plex.direct` certificate there. That name cannot cover
-# `plex-vip.${BASE_DOMAIN}`, the hostname we advertise, so without this the advertised `https://`
-# URL fails hostname verification for every client. That is not hypothetical: it is the shape of
-# the 2026-10-04 outage, where Plex's log carried a continuous storm of
-# `CERT: incomplete TLS handshake ... sslv3 alert certificate unknown` from real LAN clients while
-# plex.tv could publish nothing and the server vanished from the account.
-#
-# A SAN match is not what makes Plex FUNCTION — `httpsRequired=0` and `secureConnections` default
-# to Preferred, so clients can fall back to cleartext. It is what keeps the REMOTE leg encrypted
-# rather than silently downgrading traffic that crosses the internet.
-#
-# HOW IT REACHES PLEX. catalog/plex.yaml takes `tls.crt`/`tls.key` from this Secret as env
-# (`optional: true` — see below), and its `lifecycle.postStart` hook converts the PEM into the
-# PKCS#12 bundle Plex's `customCertificatePath` requires and sets the three `customCertificate*`
-# prefs. The conversion is redone on EVERY pod start, so a cert-manager RENEWAL is picked up with
-# no extra moving part and nothing to remember.
-#
-# ORDERING, AND WHY THE ENV REFS ARE OPTIONAL. This Certificate must live in the `plex` namespace
-# for the Deployment to read its Secret, and that namespace is created by the rendered
-# apps/wave-0/plex.yaml ITSELF — so this cannot be issued in an earlier tier (platform-config runs
-# before the namespace exists) and lands in the SAME reconcile as the workload. On a cold bootstrap
-# the Secret is therefore absent for as long as a DNS01 challenge takes. With hard env refs Plex
-# would sit in CreateContainerConfigError until then, and because `apps-wave-0` is `wait: true`
-# with `apps-wave-1` dependsOn it, a failed or slow issuance would STALL THE WHOLE APP TIER. Hence
-# `optional: true` on both refs (schema 1.11.0, added for exactly this): no certificate means Plex
-# still starts and serves on its own plex.direct cert, the hook logs that TLS was not installed,
-# and the next start after issuance converges.
-#
-# ISSUER: ClusterIssuer `letsencrypt-prod` (platform/cert-manager/clusterissuers.yaml), the same
-# issuer behind the Gateway's wildcard (platform/gateway/wildcard-certificate.yaml, Ready 28d on
-# prod as measured 2026-10-04). A SINGLE name is requested rather than reusing the wildcard,
-# because the wildcard's Secret lives in the `gateway` namespace and a Secret cannot be read across
-# namespaces; requesting `*.${BASE_DOMAIN}` again here would also burn a duplicate-certificate slot
-# against Let's Encrypt's weekly limit for no benefit. A single name is the least-privilege shape:
-# this key signs for `plex-vip.${BASE_DOMAIN}` and nothing else.
-apiVersion: cert-manager.io/v1
-kind: Certificate
-metadata:
-  name: plex-vip
-  namespace: plex
-  labels:
-    app.kubernetes.io/name: plex
-spec:
-  # The name Plex ADVERTISES and the name on this certificate must stay identical. They are two
-  # halves of one decision: catalog/plex.yaml's ADVERTISE_IP, the postStart hook's
-  # customCertificateDomain, this dnsNames entry, and catalog/pihole.yaml's split-horizon answer.
-  # Nothing enforces that they agree.
-  dnsNames:
-    - plex-vip.${BASE_DOMAIN}
-  secretName: plex-vip-tls
-  issuerRef:
-    kind: ClusterIssuer
-    name: letsencrypt-prod
@@ -138,7 +138,7 @@ spec:
              name: plex-claim
              key: PLEX_CLAIM
        - name: ADVERTISE_IP
-          value: https://plex-vip.${BASE_DOMAIN}:32400
+          value: http://plex-vip.${BASE_DOMAIN}:32400
        - name: PLEX_TLS_CRT
          valueFrom:
            secretKeyRef:
@@ -107,8 +107,42 @@ env:
  # OVERLAY fs rather than the /config PVC — so this is re-applied on EVERY pod start and git, not
  # a web form, owns the value. Hand-typed custom URLs are what broke this service; they are now
  # overwritten on every start.
+  #
+  # SCHEME IS `http` AND THAT IS TEMPORARY, 2026-10-04. It was `https`, which requires Plex to serve
+  # a certificate for this name — and the Certificate that would have supplied one is NOT APPLIED
+  # (see apps/wave-0/plex-cert.yaml's header). Advertising `https` without it reproduces the
+  # 2026-10-04 outage exactly: Plex answers 32400 with its `*.<serverhash>.plex.direct` cert, every
+  # TLS-validating client rejects the hostname, and plex.tv can publish nothing. `http` works TODAY
+  # and is MEASURED to: PiHole answers plex-vip.${BASE_DOMAIN} with ${PLEX_L4_VIP} (verified live
+  # against pihole-0, 2026-10-04) and Plex serves 32400 there.
+  #
+  # THE COST, STATED: the remote leg is unencrypted. That is tolerable only because remote access
+  # does not work yet at all (the WAN tcp/32400 forward is still absent), so today this carries LAN
+  # traffic only. RESTORE `https` IN THE SAME COMMIT that re-applies the Certificate — the two are
+  # one change, and splitting them is what broke this.
+  #
+  # HOW TO GET `https` BACK. The Certificate that would serve this name was applied on 2026-10-04
+  # and STALLED THE ENTIRE APP TIER; the file was DELETED rather than merely unregistered, because
+  # `tls-lifecycle-automated` (R-TLS-02) fails any Certificate left in the tree that no Flux path
+  # reconciles. Full post-mortem: decisions/2026-10-04-plex-unblock-app-tier.md. Three things must
+  # land TOGETHER:
+  #   1. EXEMPT cert-manager FROM `secret-provenance`. Its temporary private-key Secrets
+  #      (`plex-vip-<random>`) are denied by `secrets-must-be-flux-managed`, which matches every
+  #      Secret in a namespace labelled `homelab.keithjackson.dev/app-namespace: true` — `plex` HAS
+  #      that label, `gateway` (where the working wildcard cert lives) does NOT, which is why this
+  #      never bit the existing Certificate. `secretTemplate` cannot fix it: those Secrets come from
+  #      the keymanager controller, not the Certificate's template. Narrow fix: an
+  #      `exclude.any[].subjects` entry for ServiceAccount `cert-manager`/`cert-manager` (confirmed
+  #      live as the controller's SA). `platform/kyverno/**` is a SENSITIVE path — non-author approval.
+  #   2. GIVE THE CERTIFICATE A NON-GATING HOME. `apps-wave-0` is `wait: true`, so Flux health-checks
+  #      the Certificate OBJECT and a pending issuance stalls the whole tier
+  #      (`timeout waiting for: [Certificate/plex/plex-vip status: 'InProgress']`). `optional: true`
+  #      on the env refs below protects the POD, not the Kustomization — that distinction is the
+  #      whole bug. Use a separate Kustomization with `wait: false`, the shape
+  #      clusters/base/backup.yaml already uses for exactly this reason.
+  #   3. Flip this value back to `https`.
  - name: ADVERTISE_IP
-    value: https://plex-vip.${BASE_DOMAIN}:32400
+    value: http://plex-vip.${BASE_DOMAIN}:32400
  # THE TLS MATERIAL FOR THE NAME ABOVE, as PEM text in env rather than a mounted volume.
  # `plex-vip-tls` is issued by cert-manager from ClusterIssuer `letsencrypt-prod` for exactly
  # `plex-vip.${BASE_DOMAIN}` (apps/wave-0/plex-cert.yaml, same wave => same reconcile as this
@@ -2209,23 +2209,6 @@ EXTRA_UNITS: dict = {
            "plex-remote.yaml",
        ],
    },
-    # Plex's own TLS certificate for the single hostname it advertises to plex.tv
-    # (apps/wave-0/plex-cert.yaml). `above: False` co-locates it in plex's OWN wave, which is the
-    # only placement available: the Certificate must land in the `plex` namespace, and that
-    # namespace is created by the rendered plex.yaml in this very wave, so it CANNOT be issued
-    # earlier. The workload tolerates the resulting gap by marking the env refs `optional` rather
-    # than by ordering -- see the header of plex-cert.yaml.
-    #
-    # Non-catalog because the ServiceDefinition schema has no field for a cert-manager Certificate
-    # (additionalProperties is false); `ingressRoutes` inherits the Gateway wildcard instead, which
-    # is the opposite case to a workload terminating its own TLS on a raw L4 port.
-    "plex-cert": {
-        "anchor": "plex",
-        "above": False,
-        "files": [
-            "plex-cert.yaml",
-        ],
-    },
 }


@@ -0,0 +1,126 @@
+---
+date: 2026-10-04
+branch: fix/plex-unblock-app-tier
+title: Unblock the app tier — my Certificate stalled apps-wave-0, and the advertised https URL had no servable certificate
+---
+
+## What changed
+
+**I broke the app tier with PR #48 and this reverts the part that did it.** Two faults, both mine,
+both measured after the merge:
+
+1. **Kyverno blocks cert-manager in app namespaces.** The temporary private-key Secrets
+   (`plex-vip-<random>`) were denied by `secret-provenance/secrets-must-be-flux-managed`:
+
+       resource Secret/plex/plex-vip-rp4qj was blocked due to the following policies
+       secret-provenance:
+         secrets-must-be-flux-managed: Secrets in application namespaces must be delivered by
+         Flux from SOPS-encrypted git ... or be exempt controller-generated Secrets.
+
+   The policy matches every Secret in a namespace labelled
+   `homelab.keithjackson.dev/app-namespace: true` and excludes only Secrets labelled
+   `homelab.keithjackson.dev/secret-managed-by: controller` or carrying a Flux label. **The `plex`
+   namespace has that label; the `gateway` namespace — where the working wildcard cert lives — does
+   NOT.** That asymmetry is why the existing Certificate was never affected and mine was, and I did
+   not check it. cert-manager's temp key Secrets carry neither exempting label, and `secretTemplate`
+   cannot label them (the keymanager controller creates them, not the Certificate template). Result:
+   **no CertificateRequest, Order or Challenge was ever created.**
+
+2. **Flux health-checks the Certificate — guarding the pod was not enough.** `optional: true` did
+   exactly what it was designed to do: Plex rolled and ran fine without the Secret (verified). But
+   `apps-wave-0` is `wait: true`, so Flux waits on every applied object with a known health type:
+
+       health check failed after 5m0.045564329s: timeout waiting for:
+       [Certificate/plex/plex-vip status: 'InProgress']
+
+   apps-wave-0 never went Ready, and apps-wave-1/-2 plus everything downstream of platform-config
+   sat in `DependencyNotReady` behind it. I reasoned about the pod and missed the object.
+
+A third, independent problem the same PR introduced: **the advertised URL was `https://` with no
+servable certificate.** Plex answers 32400 with its `*.<serverhash>.plex.direct` cert, which cannot
+cover `plex-vip.${BASE_DOMAIN}` — the precise failure mode of the outage earlier that day.
+
+- **`apps/wave-0/plex-cert.yaml` DELETED**, and `plex-cert.yaml` removed from
+  `apps/wave-0/kustomization.yaml` `resources`, so Flux prunes the stuck Certificate and the health
+  gate clears. **Deleted rather than merely unregistered** — my first attempt kept the file as
+  "tracked but not onboarded" (the shape `keycloak.yaml` and `vm-demo.yaml` use) and **the repo's own
+  gate rejected it**: `tls-lifecycle-automated` (R-TLS-02) fails any Certificate in the tree that no
+  Flux path reconciles — *"nothing reconciles it, so a deleted Certificate/issuer/controller would
+  stay deleted until a human re-applied it."* That pattern is legal for a Deployment and not for a
+  TLS object, which is a distinction I had not known.
+- **`catalog/render.py`:** the now-dangling `plex-cert` EXTRA_UNITS entry removed.
+- **The re-enable recipe is recorded at the call site** — in `catalog/plex.yaml` beside the scheme
+  it is coupled to — rather than in the deleted file: the Kyverno exemption, the non-gating
+  Kustomization, and the `https` flip, as one change.
+- **`catalog/plex.yaml`:** `ADVERTISE_IP` → **`http://`**`plex-vip.${BASE_DOMAIN}:32400`, explicitly
+  temporary, with the restore condition written at the call site.
+
+**Everything else from PR #48 is kept and working** — the hostname design, the PiHole split-horizon
+answer, the `LanNetworksBandwidth` hook, the `optional` schema field, and the TLS env (inert without
+the Secret, so re-enabling is additive).
+
+## What I verified
+
+Live prod, 2026-10-04, read-only, after the #48/#50 merges.
+
+- **The stall and its exact cause**: the health-check timeout message above, and the full dependency
+  cascade (`cilium` → "dependency 'flux-system/flux-system' revision is not up to date", everything
+  else `DependencyNotReady`). The root `cilium` wait was a normal revision-gated cascade; the
+  *Certificate* was the real blocker.
+- **The Kyverno denial text, verbatim**, from the cert-manager controller log, repeating every ~30s
+  across at least 15 distinct Secret names.
+- **The policy's own structure**, read from `platform/kyverno/admission/secret-provenance.yaml`:
+  match = Secrets in `app-namespace: true` namespaces; exclude = `secret-managed-by: controller` OR
+  a `kustomize.toolkit.fluxcd.io/name` label; validate = unconditional `deny`.
+- **The namespace-label asymmetry**: `plex` carries
+  `homelab.keithjackson.dev/app-namespace: "true"`; `gateway` does not.
+- **cert-manager's controller ServiceAccount is `cert-manager`** in namespace `cert-manager` — the
+  identity a future exemption should name.
+- **`optional: true` worked**: the new pod (`plex-c4f87c84d-zgjq5`) is `1/1 Running` with no Secret
+  present, and `/config/certs` was never created.
+- **The postStart hook RAN and took effect** — the thing the whole hook was written for:
+  `LanNetworksBandwidth="10.0.0.0/8,192.168.0.0/16"` is now set in the live `Preferences.xml`, and
+  `customConnections="https://plex-vip.home.keithjackson.dev:32400"` was applied from the env. The
+  certificate prefs are correctly **ABSENT**, i.e. the hook's "no bundle → leave them alone" guard
+  behaved as designed.
+- **PiHole's split-horizon answer is correct**, asked of `pihole-0` directly:
+  `plex-vip.home.keithjackson.dev -> 192.168.2.24`, alongside `plex -> .20` and `crafty-mc -> .22`,
+  and the live `/etc/pihole/dnsmasq.conf` carries
+  `address=/plex-vip.home.keithjackson.dev/192.168.2.24`. **This corrects my own earlier check**,
+  which read `.20` and which I nearly reported as a failure — that test resolved via CLUSTER DNS
+  (CoreDNS) from inside a pod and never consulted PiHole at all.
+- **The gate caught my first attempt and I fixed it rather than working around it.** Keeping the
+  file unregistered produced `MERGE GATE: BLOCKED` (rc 1) with
+  `CHECK FAIL (tls-lifecycle-automated): 1 of 612 assertions failed`. After deleting the file and
+  the dangling EXTRA_UNITS entry: **rc 3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`,
+  `render-golden`, `l1` and `approval-policy` all **pass** — the rc 3 being only the two known
+  environmental skips (`gitea-settings`, `conformance` L2/L3). **Not green. Reported as rc 3.**
+
+## What I did NOT verify
+
+- **Whether this actually unblocks the tier.** Unreconciled at time of writing. The reasoning is
+  that pruning the Certificate removes the only failing health check, but I have not seen
+  apps-wave-0 go Ready.
+- **Whether `http://` restores client connectivity.** PiHole resolves the name and Plex serves 32400
+  (both measured), but I have not watched a Plex client complete a connection over the advertised
+  URL, and I did not re-check whether plex.tv publishes it.
+- **THE REMOTE BADGE — the symptom that started this.** `LanNetworksBandwidth` is set, but the log
+  still tags traffic `Allowed Network (WAN)`: **82 WAN, 0 LAN**, with entries continuing after the
+  pref was written. So either Plex needs a restart for it to take effect, or that log tag is not
+  driven by this pref at all. **I do not know which, and the user-visible badge is the only real
+  test** — it is in the Plex client, which I cannot see.
+- **The Hubble flow export is NOT active.** The cilium agents are still the pre-merge pods, so the
+  `filePath: stdout` question is still open and no flow has been exported. The Cilium rollout — and
+  its LAN-DNS risk, since prod-3 holds the Gateway, PiHole-DNS and Plex VIPs — is still AHEAD.
+- **Whether a cert-manager exemption is the right shape**, or whether it widens
+  `secret-provenance` more than the operator wants. It is `platform/kyverno/**`, a sensitive path,
+  and deliberately not attempted here.
+- **Whether the pruned Certificate leaves residue** — I did not check for orphaned
+  `plex-vip-<random>` Secrets or CertificateRequests after the prune. Note Kyverno DENIED every one
+  of those Secrets, so there may be nothing to clean, but that is an inference, not a check.
+- **Whether `tls-lifecycle-automated` would also have objected to the Certificate living in a
+  `wait: false` Kustomization** (the recommended re-enable shape). It demands a reconciling Flux
+  path, which that provides, but I did not test it.
+- **The unencrypted advertised URL.** `http://` is tolerable only while remote access does not work
+  (no WAN forward). Nothing enforces that assumption, and nothing will complain if `http` is still
+  there the day the port-forward appears.