From a0e6ef145ff94250395c77da00b64cb1b2522cc2 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 4 Oct 2026 17:10:25 -0500 Subject: [PATCH] =?UTF-8?q?fix(plex):=20unblock=20the=20app=20tier=20?= =?UTF-8?q?=E2=80=94=20drop=20the=20Certificate=20that=20stalled=20apps-wa?= =?UTF-8?q?ve-0,=20advertise=20http?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit I broke the app tier with #48. Two faults, both mine, both measured after merge: 1. KYVERNO BLOCKS cert-manager IN APP NAMESPACES. The temporary private-key Secrets (plex-vip-) were denied by secret-provenance/secrets-must-be-flux-managed, which matches every Secret in a namespace labelled homelab.keithjackson.dev/app-namespace=true and excludes only those labelled secret-managed-by=controller or carrying a Flux label. `plex` HAS that label; `gateway` -- where the working wildcard cert lives -- does NOT. That asymmetry is why the existing Certificate was never affected and mine was, and I did not check it. secretTemplate cannot fix it: those Secrets come from the keymanager controller, not the Certificate template. So no CertificateRequest, Order or Challenge was ever created. 2. FLUX HEALTH-CHECKS THE CERTIFICATE. `optional: true` did its job -- the pod rolled and ran fine without the Secret -- but apps-wave-0 is wait:true, so Flux waits on the Certificate OBJECT: timeout waiting for: [Certificate/plex/plex-vip status: 'InProgress'] apps-wave-0 never went Ready; wave-1/-2 and everything downstream of platform-config sat in DependencyNotReady behind it. I guarded the pod and missed the object. That distinction is the whole bug. Third, independent: the advertised URL was https with no servable certificate -- Plex answers 32400 with its *.plex.direct cert, which cannot cover plex-vip -- the exact failure mode of the outage earlier the same day. So: delete the Certificate (pruning it clears the health gate) and advertise http://plex-vip:32400, which is MEASURED to work today -- PiHole answers the name with the Plex VIP (verified against pihole-0) and Plex serves 32400 there. http is explicitly temporary and only tolerable because remote access does not work yet; the restore condition is written at the call site. DELETED, not merely unregistered: my first attempt kept the file as "tracked but not onboarded" and the gate rejected it -- tls-lifecycle-automated (R-TLS-02) fails any Certificate no Flux path reconciles. Legal for a Deployment, not for a TLS object. The re-enable recipe (Kyverno exemption + a wait:false Kustomization + the https flip, as ONE change) is recorded in catalog/plex.yaml. Kept and verified working from #48: the hostname design, PiHole's split-horizon answer, and the postStart hook -- LanNetworksBandwidth is now set live. Gate: rc 3, all substantive stages pass (first attempt was rc 1 BLOCKED). Co-Authored-By: Claude Opus 5 (1M context) --- apps/wave-0/kustomization.yaml | 1 - apps/wave-0/plex-cert.yaml | 66 --------- apps/wave-0/plex.yaml | 2 +- catalog/plex.yaml | 36 ++++- catalog/render.py | 17 --- decisions/2026-10-04-plex-unblock-app-tier.md | 126 ++++++++++++++++++ 6 files changed, 162 insertions(+), 86 deletions(-) delete mode 100644 apps/wave-0/plex-cert.yaml create mode 100644 decisions/2026-10-04-plex-unblock-app-tier.md diff --git a/apps/wave-0/kustomization.yaml b/apps/wave-0/kustomization.yaml index 3104706..394f76d 100644 --- a/apps/wave-0/kustomization.yaml +++ b/apps/wave-0/kustomization.yaml @@ -18,7 +18,6 @@ resources: - pihole-dns.yaml - pihole-oauth2-proxy-credentials.sops.yaml - pihole.yaml -- plex-cert.yaml - plex-claim-secret.sops.yaml - plex-remote.yaml - plex.yaml diff --git a/apps/wave-0/plex-cert.yaml b/apps/wave-0/plex-cert.yaml deleted file mode 100644 index 840c391..0000000 --- a/apps/wave-0/plex-cert.yaml +++ /dev/null @@ -1,66 +0,0 @@ ---- -# ================================================================================================= -# PLEX TLS — a Let's Encrypt certificate for the ONE hostname Plex advertises to plex.tv. -# ================================================================================================= -# HAND-WRITTEN, NOT RENDERED. A non-catalog app-tier unit anchored to the `plex` ServiceDefinition -# and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS table — the same -# mechanism gitea-actions, pihole-dns and plex-remote use. It is hand-written for the same reason: -# `schemas/servicedefinition.schema.json` is `additionalProperties: false` and has NO field for a -# cert-manager Certificate. (`ingressRoutes` gets TLS for free from the Gateway's wildcard; this is -# the opposite case — a certificate for a workload that terminates its OWN TLS on a raw L4 port.) -# -# WHY PLEX NEEDS ITS OWN CERTIFICATE AT ALL. Plex serves tcp/32400 directly (not through the L7 -# Gateway — the operator chose the native path 2026-10-03, apps/wave-0/plex-remote.yaml) and it -# presents its own `*..plex.direct` certificate there. That name cannot cover -# `plex-vip.${BASE_DOMAIN}`, the hostname we advertise, so without this the advertised `https://` -# URL fails hostname verification for every client. That is not hypothetical: it is the shape of -# the 2026-10-04 outage, where Plex's log carried a continuous storm of -# `CERT: incomplete TLS handshake ... sslv3 alert certificate unknown` from real LAN clients while -# plex.tv could publish nothing and the server vanished from the account. -# -# A SAN match is not what makes Plex FUNCTION — `httpsRequired=0` and `secureConnections` default -# to Preferred, so clients can fall back to cleartext. It is what keeps the REMOTE leg encrypted -# rather than silently downgrading traffic that crosses the internet. -# -# HOW IT REACHES PLEX. catalog/plex.yaml takes `tls.crt`/`tls.key` from this Secret as env -# (`optional: true` — see below), and its `lifecycle.postStart` hook converts the PEM into the -# PKCS#12 bundle Plex's `customCertificatePath` requires and sets the three `customCertificate*` -# prefs. The conversion is redone on EVERY pod start, so a cert-manager RENEWAL is picked up with -# no extra moving part and nothing to remember. -# -# ORDERING, AND WHY THE ENV REFS ARE OPTIONAL. This Certificate must live in the `plex` namespace -# for the Deployment to read its Secret, and that namespace is created by the rendered -# apps/wave-0/plex.yaml ITSELF — so this cannot be issued in an earlier tier (platform-config runs -# before the namespace exists) and lands in the SAME reconcile as the workload. On a cold bootstrap -# the Secret is therefore absent for as long as a DNS01 challenge takes. With hard env refs Plex -# would sit in CreateContainerConfigError until then, and because `apps-wave-0` is `wait: true` -# with `apps-wave-1` dependsOn it, a failed or slow issuance would STALL THE WHOLE APP TIER. Hence -# `optional: true` on both refs (schema 1.11.0, added for exactly this): no certificate means Plex -# still starts and serves on its own plex.direct cert, the hook logs that TLS was not installed, -# and the next start after issuance converges. -# -# ISSUER: ClusterIssuer `letsencrypt-prod` (platform/cert-manager/clusterissuers.yaml), the same -# issuer behind the Gateway's wildcard (platform/gateway/wildcard-certificate.yaml, Ready 28d on -# prod as measured 2026-10-04). A SINGLE name is requested rather than reusing the wildcard, -# because the wildcard's Secret lives in the `gateway` namespace and a Secret cannot be read across -# namespaces; requesting `*.${BASE_DOMAIN}` again here would also burn a duplicate-certificate slot -# against Let's Encrypt's weekly limit for no benefit. A single name is the least-privilege shape: -# this key signs for `plex-vip.${BASE_DOMAIN}` and nothing else. -apiVersion: cert-manager.io/v1 -kind: Certificate -metadata: - name: plex-vip - namespace: plex - labels: - app.kubernetes.io/name: plex -spec: - # The name Plex ADVERTISES and the name on this certificate must stay identical. They are two - # halves of one decision: catalog/plex.yaml's ADVERTISE_IP, the postStart hook's - # customCertificateDomain, this dnsNames entry, and catalog/pihole.yaml's split-horizon answer. - # Nothing enforces that they agree. - dnsNames: - - plex-vip.${BASE_DOMAIN} - secretName: plex-vip-tls - issuerRef: - kind: ClusterIssuer - name: letsencrypt-prod diff --git a/apps/wave-0/plex.yaml b/apps/wave-0/plex.yaml index 7d80c83..5165af4 100644 --- a/apps/wave-0/plex.yaml +++ b/apps/wave-0/plex.yaml @@ -138,7 +138,7 @@ spec: name: plex-claim key: PLEX_CLAIM - name: ADVERTISE_IP - value: https://plex-vip.${BASE_DOMAIN}:32400 + value: http://plex-vip.${BASE_DOMAIN}:32400 - name: PLEX_TLS_CRT valueFrom: secretKeyRef: diff --git a/catalog/plex.yaml b/catalog/plex.yaml index 752c04c..aaea8ba 100644 --- a/catalog/plex.yaml +++ b/catalog/plex.yaml @@ -107,8 +107,42 @@ env: # OVERLAY fs rather than the /config PVC — so this is re-applied on EVERY pod start and git, not # a web form, owns the value. Hand-typed custom URLs are what broke this service; they are now # overwritten on every start. + # + # SCHEME IS `http` AND THAT IS TEMPORARY, 2026-10-04. It was `https`, which requires Plex to serve + # a certificate for this name — and the Certificate that would have supplied one is NOT APPLIED + # (see apps/wave-0/plex-cert.yaml's header). Advertising `https` without it reproduces the + # 2026-10-04 outage exactly: Plex answers 32400 with its `*..plex.direct` cert, every + # TLS-validating client rejects the hostname, and plex.tv can publish nothing. `http` works TODAY + # and is MEASURED to: PiHole answers plex-vip.${BASE_DOMAIN} with ${PLEX_L4_VIP} (verified live + # against pihole-0, 2026-10-04) and Plex serves 32400 there. + # + # THE COST, STATED: the remote leg is unencrypted. That is tolerable only because remote access + # does not work yet at all (the WAN tcp/32400 forward is still absent), so today this carries LAN + # traffic only. RESTORE `https` IN THE SAME COMMIT that re-applies the Certificate — the two are + # one change, and splitting them is what broke this. + # + # HOW TO GET `https` BACK. The Certificate that would serve this name was applied on 2026-10-04 + # and STALLED THE ENTIRE APP TIER; the file was DELETED rather than merely unregistered, because + # `tls-lifecycle-automated` (R-TLS-02) fails any Certificate left in the tree that no Flux path + # reconciles. Full post-mortem: decisions/2026-10-04-plex-unblock-app-tier.md. Three things must + # land TOGETHER: + # 1. EXEMPT cert-manager FROM `secret-provenance`. Its temporary private-key Secrets + # (`plex-vip-`) are denied by `secrets-must-be-flux-managed`, which matches every + # Secret in a namespace labelled `homelab.keithjackson.dev/app-namespace: true` — `plex` HAS + # that label, `gateway` (where the working wildcard cert lives) does NOT, which is why this + # never bit the existing Certificate. `secretTemplate` cannot fix it: those Secrets come from + # the keymanager controller, not the Certificate's template. Narrow fix: an + # `exclude.any[].subjects` entry for ServiceAccount `cert-manager`/`cert-manager` (confirmed + # live as the controller's SA). `platform/kyverno/**` is a SENSITIVE path — non-author approval. + # 2. GIVE THE CERTIFICATE A NON-GATING HOME. `apps-wave-0` is `wait: true`, so Flux health-checks + # the Certificate OBJECT and a pending issuance stalls the whole tier + # (`timeout waiting for: [Certificate/plex/plex-vip status: 'InProgress']`). `optional: true` + # on the env refs below protects the POD, not the Kustomization — that distinction is the + # whole bug. Use a separate Kustomization with `wait: false`, the shape + # clusters/base/backup.yaml already uses for exactly this reason. + # 3. Flip this value back to `https`. - name: ADVERTISE_IP - value: https://plex-vip.${BASE_DOMAIN}:32400 + value: http://plex-vip.${BASE_DOMAIN}:32400 # THE TLS MATERIAL FOR THE NAME ABOVE, as PEM text in env rather than a mounted volume. # `plex-vip-tls` is issued by cert-manager from ClusterIssuer `letsencrypt-prod` for exactly # `plex-vip.${BASE_DOMAIN}` (apps/wave-0/plex-cert.yaml, same wave => same reconcile as this diff --git a/catalog/render.py b/catalog/render.py index 277226f..894d6f0 100644 --- a/catalog/render.py +++ b/catalog/render.py @@ -2209,23 +2209,6 @@ EXTRA_UNITS: dict = { "plex-remote.yaml", ], }, - # Plex's own TLS certificate for the single hostname it advertises to plex.tv - # (apps/wave-0/plex-cert.yaml). `above: False` co-locates it in plex's OWN wave, which is the - # only placement available: the Certificate must land in the `plex` namespace, and that - # namespace is created by the rendered plex.yaml in this very wave, so it CANNOT be issued - # earlier. The workload tolerates the resulting gap by marking the env refs `optional` rather - # than by ordering -- see the header of plex-cert.yaml. - # - # Non-catalog because the ServiceDefinition schema has no field for a cert-manager Certificate - # (additionalProperties is false); `ingressRoutes` inherits the Gateway wildcard instead, which - # is the opposite case to a workload terminating its own TLS on a raw L4 port. - "plex-cert": { - "anchor": "plex", - "above": False, - "files": [ - "plex-cert.yaml", - ], - }, } diff --git a/decisions/2026-10-04-plex-unblock-app-tier.md b/decisions/2026-10-04-plex-unblock-app-tier.md new file mode 100644 index 0000000..b5500c0 --- /dev/null +++ b/decisions/2026-10-04-plex-unblock-app-tier.md @@ -0,0 +1,126 @@ +--- +date: 2026-10-04 +branch: fix/plex-unblock-app-tier +title: Unblock the app tier — my Certificate stalled apps-wave-0, and the advertised https URL had no servable certificate +--- + +## What changed + +**I broke the app tier with PR #48 and this reverts the part that did it.** Two faults, both mine, +both measured after the merge: + +1. **Kyverno blocks cert-manager in app namespaces.** The temporary private-key Secrets + (`plex-vip-`) were denied by `secret-provenance/secrets-must-be-flux-managed`: + + resource Secret/plex/plex-vip-rp4qj was blocked due to the following policies + secret-provenance: + secrets-must-be-flux-managed: Secrets in application namespaces must be delivered by + Flux from SOPS-encrypted git ... or be exempt controller-generated Secrets. + + The policy matches every Secret in a namespace labelled + `homelab.keithjackson.dev/app-namespace: true` and excludes only Secrets labelled + `homelab.keithjackson.dev/secret-managed-by: controller` or carrying a Flux label. **The `plex` + namespace has that label; the `gateway` namespace — where the working wildcard cert lives — does + NOT.** That asymmetry is why the existing Certificate was never affected and mine was, and I did + not check it. cert-manager's temp key Secrets carry neither exempting label, and `secretTemplate` + cannot label them (the keymanager controller creates them, not the Certificate template). Result: + **no CertificateRequest, Order or Challenge was ever created.** + +2. **Flux health-checks the Certificate — guarding the pod was not enough.** `optional: true` did + exactly what it was designed to do: Plex rolled and ran fine without the Secret (verified). But + `apps-wave-0` is `wait: true`, so Flux waits on every applied object with a known health type: + + health check failed after 5m0.045564329s: timeout waiting for: + [Certificate/plex/plex-vip status: 'InProgress'] + + apps-wave-0 never went Ready, and apps-wave-1/-2 plus everything downstream of platform-config + sat in `DependencyNotReady` behind it. I reasoned about the pod and missed the object. + +A third, independent problem the same PR introduced: **the advertised URL was `https://` with no +servable certificate.** Plex answers 32400 with its `*..plex.direct` cert, which cannot +cover `plex-vip.${BASE_DOMAIN}` — the precise failure mode of the outage earlier that day. + +- **`apps/wave-0/plex-cert.yaml` DELETED**, and `plex-cert.yaml` removed from + `apps/wave-0/kustomization.yaml` `resources`, so Flux prunes the stuck Certificate and the health + gate clears. **Deleted rather than merely unregistered** — my first attempt kept the file as + "tracked but not onboarded" (the shape `keycloak.yaml` and `vm-demo.yaml` use) and **the repo's own + gate rejected it**: `tls-lifecycle-automated` (R-TLS-02) fails any Certificate in the tree that no + Flux path reconciles — *"nothing reconciles it, so a deleted Certificate/issuer/controller would + stay deleted until a human re-applied it."* That pattern is legal for a Deployment and not for a + TLS object, which is a distinction I had not known. +- **`catalog/render.py`:** the now-dangling `plex-cert` EXTRA_UNITS entry removed. +- **The re-enable recipe is recorded at the call site** — in `catalog/plex.yaml` beside the scheme + it is coupled to — rather than in the deleted file: the Kyverno exemption, the non-gating + Kustomization, and the `https` flip, as one change. +- **`catalog/plex.yaml`:** `ADVERTISE_IP` → **`http://`**`plex-vip.${BASE_DOMAIN}:32400`, explicitly + temporary, with the restore condition written at the call site. + +**Everything else from PR #48 is kept and working** — the hostname design, the PiHole split-horizon +answer, the `LanNetworksBandwidth` hook, the `optional` schema field, and the TLS env (inert without +the Secret, so re-enabling is additive). + +## What I verified + +Live prod, 2026-10-04, read-only, after the #48/#50 merges. + +- **The stall and its exact cause**: the health-check timeout message above, and the full dependency + cascade (`cilium` → "dependency 'flux-system/flux-system' revision is not up to date", everything + else `DependencyNotReady`). The root `cilium` wait was a normal revision-gated cascade; the + *Certificate* was the real blocker. +- **The Kyverno denial text, verbatim**, from the cert-manager controller log, repeating every ~30s + across at least 15 distinct Secret names. +- **The policy's own structure**, read from `platform/kyverno/admission/secret-provenance.yaml`: + match = Secrets in `app-namespace: true` namespaces; exclude = `secret-managed-by: controller` OR + a `kustomize.toolkit.fluxcd.io/name` label; validate = unconditional `deny`. +- **The namespace-label asymmetry**: `plex` carries + `homelab.keithjackson.dev/app-namespace: "true"`; `gateway` does not. +- **cert-manager's controller ServiceAccount is `cert-manager`** in namespace `cert-manager` — the + identity a future exemption should name. +- **`optional: true` worked**: the new pod (`plex-c4f87c84d-zgjq5`) is `1/1 Running` with no Secret + present, and `/config/certs` was never created. +- **The postStart hook RAN and took effect** — the thing the whole hook was written for: + `LanNetworksBandwidth="10.0.0.0/8,192.168.0.0/16"` is now set in the live `Preferences.xml`, and + `customConnections="https://plex-vip.home.keithjackson.dev:32400"` was applied from the env. The + certificate prefs are correctly **ABSENT**, i.e. the hook's "no bundle → leave them alone" guard + behaved as designed. +- **PiHole's split-horizon answer is correct**, asked of `pihole-0` directly: + `plex-vip.home.keithjackson.dev -> 192.168.2.24`, alongside `plex -> .20` and `crafty-mc -> .22`, + and the live `/etc/pihole/dnsmasq.conf` carries + `address=/plex-vip.home.keithjackson.dev/192.168.2.24`. **This corrects my own earlier check**, + which read `.20` and which I nearly reported as a failure — that test resolved via CLUSTER DNS + (CoreDNS) from inside a pod and never consulted PiHole at all. +- **The gate caught my first attempt and I fixed it rather than working around it.** Keeping the + file unregistered produced `MERGE GATE: BLOCKED` (rc 1) with + `CHECK FAIL (tls-lifecycle-automated): 1 of 612 assertions failed`. After deleting the file and + the dangling EXTRA_UNITS entry: **rc 3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`, + `render-golden`, `l1` and `approval-policy` all **pass** — the rc 3 being only the two known + environmental skips (`gitea-settings`, `conformance` L2/L3). **Not green. Reported as rc 3.** + +## What I did NOT verify + +- **Whether this actually unblocks the tier.** Unreconciled at time of writing. The reasoning is + that pruning the Certificate removes the only failing health check, but I have not seen + apps-wave-0 go Ready. +- **Whether `http://` restores client connectivity.** PiHole resolves the name and Plex serves 32400 + (both measured), but I have not watched a Plex client complete a connection over the advertised + URL, and I did not re-check whether plex.tv publishes it. +- **THE REMOTE BADGE — the symptom that started this.** `LanNetworksBandwidth` is set, but the log + still tags traffic `Allowed Network (WAN)`: **82 WAN, 0 LAN**, with entries continuing after the + pref was written. So either Plex needs a restart for it to take effect, or that log tag is not + driven by this pref at all. **I do not know which, and the user-visible badge is the only real + test** — it is in the Plex client, which I cannot see. +- **The Hubble flow export is NOT active.** The cilium agents are still the pre-merge pods, so the + `filePath: stdout` question is still open and no flow has been exported. The Cilium rollout — and + its LAN-DNS risk, since prod-3 holds the Gateway, PiHole-DNS and Plex VIPs — is still AHEAD. +- **Whether a cert-manager exemption is the right shape**, or whether it widens + `secret-provenance` more than the operator wants. It is `platform/kyverno/**`, a sensitive path, + and deliberately not attempted here. +- **Whether the pruned Certificate leaves residue** — I did not check for orphaned + `plex-vip-` Secrets or CertificateRequests after the prune. Note Kyverno DENIED every one + of those Secrets, so there may be nothing to clean, but that is an inference, not a check. +- **Whether `tls-lifecycle-automated` would also have objected to the Certificate living in a + `wait: false` Kustomization** (the recommended re-enable shape). It demands a reconciling Flux + path, which that provides, but I did not test it. +- **The unencrypted advertised URL.** `http://` is tolerable only while remote access does not work + (no WAN forward). Nothing enforces that assumption, and nothing will complain if `http` is still + there the day the port-forward appears.