Merge pull request 'fix(plex): unblock the app tier — drop the Certificate that stalled apps-wave-0, advertise http' (#51) from fix/plex-unblock-app-tier into main

Reviewed-on: #51
2026-10-05 13:53:21 +00:00
commit bc7095677b
@@ -18,7 +18,6 @@ resources:
 - pihole-dns.yaml
 - pihole-oauth2-proxy-credentials.sops.yaml
 - pihole.yaml
 - plex-cert.yaml
 - plex-claim-secret.sops.yaml
 - plex-remote.yaml
 - plex.yaml
@@ -1,66 +0,0 @@
 ---
 # =================================================================================================
 # PLEX TLS — a Let's Encrypt certificate for the ONE hostname Plex advertises to plex.tv.
 # =================================================================================================
 # HAND-WRITTEN, NOT RENDERED. A non-catalog app-tier unit anchored to the `plex` ServiceDefinition
 # and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS table — the same
 # mechanism gitea-actions, pihole-dns and plex-remote use. It is hand-written for the same reason:
 # `schemas/servicedefinition.schema.json` is `additionalProperties: false` and has NO field for a
 # cert-manager Certificate. (`ingressRoutes` gets TLS for free from the Gateway's wildcard; this is
 # the opposite case — a certificate for a workload that terminates its OWN TLS on a raw L4 port.)
 #
 # WHY PLEX NEEDS ITS OWN CERTIFICATE AT ALL. Plex serves tcp/32400 directly (not through the L7
 # Gateway — the operator chose the native path 2026-10-03, apps/wave-0/plex-remote.yaml) and it
 # presents its own `*.<serverhash>.plex.direct` certificate there. That name cannot cover
 # `plex-vip.${BASE_DOMAIN}`, the hostname we advertise, so without this the advertised `https://`
 # URL fails hostname verification for every client. That is not hypothetical: it is the shape of
 # the 2026-10-04 outage, where Plex's log carried a continuous storm of
 # `CERT: incomplete TLS handshake ... sslv3 alert certificate unknown` from real LAN clients while
 # plex.tv could publish nothing and the server vanished from the account.
 #
 # A SAN match is not what makes Plex FUNCTION — `httpsRequired=0` and `secureConnections` default
 # to Preferred, so clients can fall back to cleartext. It is what keeps the REMOTE leg encrypted
 # rather than silently downgrading traffic that crosses the internet.
 #
 # HOW IT REACHES PLEX. catalog/plex.yaml takes `tls.crt`/`tls.key` from this Secret as env
 # (`optional: true` — see below), and its `lifecycle.postStart` hook converts the PEM into the
 # PKCS#12 bundle Plex's `customCertificatePath` requires and sets the three `customCertificate*`
 # prefs. The conversion is redone on EVERY pod start, so a cert-manager RENEWAL is picked up with
 # no extra moving part and nothing to remember.
 #
 # ORDERING, AND WHY THE ENV REFS ARE OPTIONAL. This Certificate must live in the `plex` namespace
 # for the Deployment to read its Secret, and that namespace is created by the rendered
 # apps/wave-0/plex.yaml ITSELF — so this cannot be issued in an earlier tier (platform-config runs
 # before the namespace exists) and lands in the SAME reconcile as the workload. On a cold bootstrap
 # the Secret is therefore absent for as long as a DNS01 challenge takes. With hard env refs Plex
 # would sit in CreateContainerConfigError until then, and because `apps-wave-0` is `wait: true`
 # with `apps-wave-1` dependsOn it, a failed or slow issuance would STALL THE WHOLE APP TIER. Hence
 # `optional: true` on both refs (schema 1.11.0, added for exactly this): no certificate means Plex
 # still starts and serves on its own plex.direct cert, the hook logs that TLS was not installed,
 # and the next start after issuance converges.
 #
 # ISSUER: ClusterIssuer `letsencrypt-prod` (platform/cert-manager/clusterissuers.yaml), the same
 # issuer behind the Gateway's wildcard (platform/gateway/wildcard-certificate.yaml, Ready 28d on
 # prod as measured 2026-10-04). A SINGLE name is requested rather than reusing the wildcard,
 # because the wildcard's Secret lives in the `gateway` namespace and a Secret cannot be read across
 # namespaces; requesting `*.${BASE_DOMAIN}` again here would also burn a duplicate-certificate slot
 # against Let's Encrypt's weekly limit for no benefit. A single name is the least-privilege shape:
 # this key signs for `plex-vip.${BASE_DOMAIN}` and nothing else.
 apiVersion: cert-manager.io/v1
 kind: Certificate
 metadata:
  name: plex-vip
  namespace: plex
  labels:
    app.kubernetes.io/name: plex
 spec:
  # The name Plex ADVERTISES and the name on this certificate must stay identical. They are two
  # halves of one decision: catalog/plex.yaml's ADVERTISE_IP, the postStart hook's
  # customCertificateDomain, this dnsNames entry, and catalog/pihole.yaml's split-horizon answer.
  # Nothing enforces that they agree.
  dnsNames:
    - plex-vip.${BASE_DOMAIN}
  secretName: plex-vip-tls
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-prod
@@ -138,7 +138,7 @@ spec:
              name: plex-claim
              key: PLEX_CLAIM
        - name: ADVERTISE_IP
-          value: https://plex-vip.${BASE_DOMAIN}:32400
+          value: http://plex-vip.${BASE_DOMAIN}:32400
        - name: PLEX_TLS_CRT
          valueFrom:
            secretKeyRef:
@@ -107,8 +107,42 @@ env:
  # OVERLAY fs rather than the /config PVC — so this is re-applied on EVERY pod start and git, not
  # a web form, owns the value. Hand-typed custom URLs are what broke this service; they are now
  # overwritten on every start.
  #
  # SCHEME IS `http` AND THAT IS TEMPORARY, 2026-10-04. It was `https`, which requires Plex to serve
  # a certificate for this name — and the Certificate that would have supplied one is NOT APPLIED
  # (see apps/wave-0/plex-cert.yaml's header). Advertising `https` without it reproduces the
  # 2026-10-04 outage exactly: Plex answers 32400 with its `*.<serverhash>.plex.direct` cert, every
  # TLS-validating client rejects the hostname, and plex.tv can publish nothing. `http` works TODAY
  # and is MEASURED to: PiHole answers plex-vip.${BASE_DOMAIN} with ${PLEX_L4_VIP} (verified live
  # against pihole-0, 2026-10-04) and Plex serves 32400 there.
  #
  # THE COST, STATED: the remote leg is unencrypted. That is tolerable only because remote access
  # does not work yet at all (the WAN tcp/32400 forward is still absent), so today this carries LAN
  # traffic only. RESTORE `https` IN THE SAME COMMIT that re-applies the Certificate — the two are
  # one change, and splitting them is what broke this.
  #
  # HOW TO GET `https` BACK. The Certificate that would serve this name was applied on 2026-10-04
  # and STALLED THE ENTIRE APP TIER; the file was DELETED rather than merely unregistered, because
  # `tls-lifecycle-automated` (R-TLS-02) fails any Certificate left in the tree that no Flux path
  # reconciles. Full post-mortem: decisions/2026-10-04-plex-unblock-app-tier.md. Three things must
  # land TOGETHER:
  #   1. EXEMPT cert-manager FROM `secret-provenance`. Its temporary private-key Secrets
  #      (`plex-vip-<random>`) are denied by `secrets-must-be-flux-managed`, which matches every
  #      Secret in a namespace labelled `homelab.keithjackson.dev/app-namespace: true` — `plex` HAS
  #      that label, `gateway` (where the working wildcard cert lives) does NOT, which is why this
  #      never bit the existing Certificate. `secretTemplate` cannot fix it: those Secrets come from
  #      the keymanager controller, not the Certificate's template. Narrow fix: an
  #      `exclude.any[].subjects` entry for ServiceAccount `cert-manager`/`cert-manager` (confirmed
  #      live as the controller's SA). `platform/kyverno/**` is a SENSITIVE path — non-author approval.
  #   2. GIVE THE CERTIFICATE A NON-GATING HOME. `apps-wave-0` is `wait: true`, so Flux health-checks
  #      the Certificate OBJECT and a pending issuance stalls the whole tier
  #      (`timeout waiting for: [Certificate/plex/plex-vip status: 'InProgress']`). `optional: true`
  #      on the env refs below protects the POD, not the Kustomization — that distinction is the
  #      whole bug. Use a separate Kustomization with `wait: false`, the shape
  #      clusters/base/backup.yaml already uses for exactly this reason.
  #   3. Flip this value back to `https`.
  - name: ADVERTISE_IP
-    value: https://plex-vip.${BASE_DOMAIN}:32400
+    value: http://plex-vip.${BASE_DOMAIN}:32400
  # THE TLS MATERIAL FOR THE NAME ABOVE, as PEM text in env rather than a mounted volume.
  # `plex-vip-tls` is issued by cert-manager from ClusterIssuer `letsencrypt-prod` for exactly
  # `plex-vip.${BASE_DOMAIN}` (apps/wave-0/plex-cert.yaml, same wave => same reconcile as this
@@ -2209,23 +2209,6 @@ EXTRA_UNITS: dict = {
            "plex-remote.yaml",
        ],
    },
    # Plex's own TLS certificate for the single hostname it advertises to plex.tv
    # (apps/wave-0/plex-cert.yaml). `above: False` co-locates it in plex's OWN wave, which is the
    # only placement available: the Certificate must land in the `plex` namespace, and that
    # namespace is created by the rendered plex.yaml in this very wave, so it CANNOT be issued
    # earlier. The workload tolerates the resulting gap by marking the env refs `optional` rather
    # than by ordering -- see the header of plex-cert.yaml.
    #
    # Non-catalog because the ServiceDefinition schema has no field for a cert-manager Certificate
    # (additionalProperties is false); `ingressRoutes` inherits the Gateway wildcard instead, which
    # is the opposite case to a workload terminating its own TLS on a raw L4 port.
    "plex-cert": {
        "anchor": "plex",
        "above": False,
        "files": [
            "plex-cert.yaml",
        ],
    },
 }
@@ -0,0 +1,126 @@
 ---
 date: 2026-10-04
 branch: fix/plex-unblock-app-tier
 title: Unblock the app tier — my Certificate stalled apps-wave-0, and the advertised https URL had no servable certificate
 ---
 ## What changed
 **I broke the app tier with PR #48 and this reverts the part that did it.** Two faults, both mine,
 both measured after the merge:
 1. **Kyverno blocks cert-manager in app namespaces.** The temporary private-key Secrets
   (`plex-vip-<random>`) were denied by `secret-provenance/secrets-must-be-flux-managed`:
       resource Secret/plex/plex-vip-rp4qj was blocked due to the following policies
       secret-provenance:
         secrets-must-be-flux-managed: Secrets in application namespaces must be delivered by
         Flux from SOPS-encrypted git ... or be exempt controller-generated Secrets.
   The policy matches every Secret in a namespace labelled
   `homelab.keithjackson.dev/app-namespace: true` and excludes only Secrets labelled
   `homelab.keithjackson.dev/secret-managed-by: controller` or carrying a Flux label. **The `plex`
   namespace has that label; the `gateway` namespace — where the working wildcard cert lives — does
   NOT.** That asymmetry is why the existing Certificate was never affected and mine was, and I did
   not check it. cert-manager's temp key Secrets carry neither exempting label, and `secretTemplate`
   cannot label them (the keymanager controller creates them, not the Certificate template). Result:
   **no CertificateRequest, Order or Challenge was ever created.**
 2. **Flux health-checks the Certificate — guarding the pod was not enough.** `optional: true` did
   exactly what it was designed to do: Plex rolled and ran fine without the Secret (verified). But
   `apps-wave-0` is `wait: true`, so Flux waits on every applied object with a known health type:
       health check failed after 5m0.045564329s: timeout waiting for:
       [Certificate/plex/plex-vip status: 'InProgress']
   apps-wave-0 never went Ready, and apps-wave-1/-2 plus everything downstream of platform-config
   sat in `DependencyNotReady` behind it. I reasoned about the pod and missed the object.
 A third, independent problem the same PR introduced: **the advertised URL was `https://` with no
 servable certificate.** Plex answers 32400 with its `*.<serverhash>.plex.direct` cert, which cannot
 cover `plex-vip.${BASE_DOMAIN}` — the precise failure mode of the outage earlier that day.
 - **`apps/wave-0/plex-cert.yaml` DELETED**, and `plex-cert.yaml` removed from
  `apps/wave-0/kustomization.yaml` `resources`, so Flux prunes the stuck Certificate and the health
  gate clears. **Deleted rather than merely unregistered** — my first attempt kept the file as
  "tracked but not onboarded" (the shape `keycloak.yaml` and `vm-demo.yaml` use) and **the repo's own
  gate rejected it**: `tls-lifecycle-automated` (R-TLS-02) fails any Certificate in the tree that no
  Flux path reconciles — *"nothing reconciles it, so a deleted Certificate/issuer/controller would
  stay deleted until a human re-applied it."* That pattern is legal for a Deployment and not for a
  TLS object, which is a distinction I had not known.
 - **`catalog/render.py`:** the now-dangling `plex-cert` EXTRA_UNITS entry removed.
 - **The re-enable recipe is recorded at the call site** — in `catalog/plex.yaml` beside the scheme
  it is coupled to — rather than in the deleted file: the Kyverno exemption, the non-gating
  Kustomization, and the `https` flip, as one change.
 - **`catalog/plex.yaml`:** `ADVERTISE_IP` → **`http://`**`plex-vip.${BASE_DOMAIN}:32400`, explicitly
  temporary, with the restore condition written at the call site.
 **Everything else from PR #48 is kept and working** — the hostname design, the PiHole split-horizon
 answer, the `LanNetworksBandwidth` hook, the `optional` schema field, and the TLS env (inert without
 the Secret, so re-enabling is additive).
 ## What I verified
 Live prod, 2026-10-04, read-only, after the #48/#50 merges.
 - **The stall and its exact cause**: the health-check timeout message above, and the full dependency
  cascade (`cilium` → "dependency 'flux-system/flux-system' revision is not up to date", everything
  else `DependencyNotReady`). The root `cilium` wait was a normal revision-gated cascade; the
  *Certificate* was the real blocker.
 - **The Kyverno denial text, verbatim**, from the cert-manager controller log, repeating every ~30s
  across at least 15 distinct Secret names.
 - **The policy's own structure**, read from `platform/kyverno/admission/secret-provenance.yaml`:
  match = Secrets in `app-namespace: true` namespaces; exclude = `secret-managed-by: controller` OR
  a `kustomize.toolkit.fluxcd.io/name` label; validate = unconditional `deny`.
 - **The namespace-label asymmetry**: `plex` carries
  `homelab.keithjackson.dev/app-namespace: "true"`; `gateway` does not.
 - **cert-manager's controller ServiceAccount is `cert-manager`** in namespace `cert-manager` — the
  identity a future exemption should name.
 - **`optional: true` worked**: the new pod (`plex-c4f87c84d-zgjq5`) is `1/1 Running` with no Secret
  present, and `/config/certs` was never created.
 - **The postStart hook RAN and took effect** — the thing the whole hook was written for:
  `LanNetworksBandwidth="10.0.0.0/8,192.168.0.0/16"` is now set in the live `Preferences.xml`, and
  `customConnections="https://plex-vip.home.keithjackson.dev:32400"` was applied from the env. The
  certificate prefs are correctly **ABSENT**, i.e. the hook's "no bundle → leave them alone" guard
  behaved as designed.
 - **PiHole's split-horizon answer is correct**, asked of `pihole-0` directly:
  `plex-vip.home.keithjackson.dev -> 192.168.2.24`, alongside `plex -> .20` and `crafty-mc -> .22`,
  and the live `/etc/pihole/dnsmasq.conf` carries
  `address=/plex-vip.home.keithjackson.dev/192.168.2.24`. **This corrects my own earlier check**,
  which read `.20` and which I nearly reported as a failure — that test resolved via CLUSTER DNS
  (CoreDNS) from inside a pod and never consulted PiHole at all.
 - **The gate caught my first attempt and I fixed it rather than working around it.** Keeping the
  file unregistered produced `MERGE GATE: BLOCKED` (rc 1) with
  `CHECK FAIL (tls-lifecycle-automated): 1 of 612 assertions failed`. After deleting the file and
  the dangling EXTRA_UNITS entry: **rc 3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`,
  `render-golden`, `l1` and `approval-policy` all **pass** — the rc 3 being only the two known
  environmental skips (`gitea-settings`, `conformance` L2/L3). **Not green. Reported as rc 3.**
 ## What I did NOT verify
 - **Whether this actually unblocks the tier.** Unreconciled at time of writing. The reasoning is
  that pruning the Certificate removes the only failing health check, but I have not seen
  apps-wave-0 go Ready.
 - **Whether `http://` restores client connectivity.** PiHole resolves the name and Plex serves 32400
  (both measured), but I have not watched a Plex client complete a connection over the advertised
  URL, and I did not re-check whether plex.tv publishes it.
 - **THE REMOTE BADGE — the symptom that started this.** `LanNetworksBandwidth` is set, but the log
  still tags traffic `Allowed Network (WAN)`: **82 WAN, 0 LAN**, with entries continuing after the
  pref was written. So either Plex needs a restart for it to take effect, or that log tag is not
  driven by this pref at all. **I do not know which, and the user-visible badge is the only real
  test** — it is in the Plex client, which I cannot see.
 - **The Hubble flow export is NOT active.** The cilium agents are still the pre-merge pods, so the
  `filePath: stdout` question is still open and no flow has been exported. The Cilium rollout — and
  its LAN-DNS risk, since prod-3 holds the Gateway, PiHole-DNS and Plex VIPs — is still AHEAD.
 - **Whether a cert-manager exemption is the right shape**, or whether it widens
  `secret-provenance` more than the operator wants. It is `platform/kyverno/**`, a sensitive path,
  and deliberately not attempted here.
 - **Whether the pruned Certificate leaves residue** — I did not check for orphaned
  `plex-vip-<random>` Secrets or CertificateRequests after the prune. Note Kyverno DENIED every one
  of those Secrets, so there may be nothing to clean, but that is an inference, not a check.
 - **Whether `tls-lifecycle-automated` would also have objected to the Certificate living in a
  `wait: false` Kustomization** (the recommended re-enable shape). It demands a reconciling Flux
  path, which that provides, but I did not test it.
 - **The unencrypted advertised URL.** `http://` is tolerable only while remote access does not work
  (no WAN forward). Nothing enforces that assumption, and nothing will complain if `http` is still
  there the day the port-forward appears.