--- date: 2026-10-04 branch: fix/plex-unblock-app-tier title: Unblock the app tier — my Certificate stalled apps-wave-0, and the advertised https URL had no servable certificate --- ## What changed **I broke the app tier with PR #48 and this reverts the part that did it.** Two faults, both mine, both measured after the merge: 1. **Kyverno blocks cert-manager in app namespaces.** The temporary private-key Secrets (`plex-vip-`) were denied by `secret-provenance/secrets-must-be-flux-managed`: resource Secret/plex/plex-vip-rp4qj was blocked due to the following policies secret-provenance: secrets-must-be-flux-managed: Secrets in application namespaces must be delivered by Flux from SOPS-encrypted git ... or be exempt controller-generated Secrets. The policy matches every Secret in a namespace labelled `homelab.keithjackson.dev/app-namespace: true` and excludes only Secrets labelled `homelab.keithjackson.dev/secret-managed-by: controller` or carrying a Flux label. **The `plex` namespace has that label; the `gateway` namespace — where the working wildcard cert lives — does NOT.** That asymmetry is why the existing Certificate was never affected and mine was, and I did not check it. cert-manager's temp key Secrets carry neither exempting label, and `secretTemplate` cannot label them (the keymanager controller creates them, not the Certificate template). Result: **no CertificateRequest, Order or Challenge was ever created.** 2. **Flux health-checks the Certificate — guarding the pod was not enough.** `optional: true` did exactly what it was designed to do: Plex rolled and ran fine without the Secret (verified). But `apps-wave-0` is `wait: true`, so Flux waits on every applied object with a known health type: health check failed after 5m0.045564329s: timeout waiting for: [Certificate/plex/plex-vip status: 'InProgress'] apps-wave-0 never went Ready, and apps-wave-1/-2 plus everything downstream of platform-config sat in `DependencyNotReady` behind it. I reasoned about the pod and missed the object. A third, independent problem the same PR introduced: **the advertised URL was `https://` with no servable certificate.** Plex answers 32400 with its `*..plex.direct` cert, which cannot cover `plex-vip.${BASE_DOMAIN}` — the precise failure mode of the outage earlier that day. - **`apps/wave-0/plex-cert.yaml` DELETED**, and `plex-cert.yaml` removed from `apps/wave-0/kustomization.yaml` `resources`, so Flux prunes the stuck Certificate and the health gate clears. **Deleted rather than merely unregistered** — my first attempt kept the file as "tracked but not onboarded" (the shape `keycloak.yaml` and `vm-demo.yaml` use) and **the repo's own gate rejected it**: `tls-lifecycle-automated` (R-TLS-02) fails any Certificate in the tree that no Flux path reconciles — *"nothing reconciles it, so a deleted Certificate/issuer/controller would stay deleted until a human re-applied it."* That pattern is legal for a Deployment and not for a TLS object, which is a distinction I had not known. - **`catalog/render.py`:** the now-dangling `plex-cert` EXTRA_UNITS entry removed. - **The re-enable recipe is recorded at the call site** — in `catalog/plex.yaml` beside the scheme it is coupled to — rather than in the deleted file: the Kyverno exemption, the non-gating Kustomization, and the `https` flip, as one change. - **`catalog/plex.yaml`:** `ADVERTISE_IP` → **`http://`**`plex-vip.${BASE_DOMAIN}:32400`, explicitly temporary, with the restore condition written at the call site. **Everything else from PR #48 is kept and working** — the hostname design, the PiHole split-horizon answer, the `LanNetworksBandwidth` hook, the `optional` schema field, and the TLS env (inert without the Secret, so re-enabling is additive). ## What I verified Live prod, 2026-10-04, read-only, after the #48/#50 merges. - **The stall and its exact cause**: the health-check timeout message above, and the full dependency cascade (`cilium` → "dependency 'flux-system/flux-system' revision is not up to date", everything else `DependencyNotReady`). The root `cilium` wait was a normal revision-gated cascade; the *Certificate* was the real blocker. - **The Kyverno denial text, verbatim**, from the cert-manager controller log, repeating every ~30s across at least 15 distinct Secret names. - **The policy's own structure**, read from `platform/kyverno/admission/secret-provenance.yaml`: match = Secrets in `app-namespace: true` namespaces; exclude = `secret-managed-by: controller` OR a `kustomize.toolkit.fluxcd.io/name` label; validate = unconditional `deny`. - **The namespace-label asymmetry**: `plex` carries `homelab.keithjackson.dev/app-namespace: "true"`; `gateway` does not. - **cert-manager's controller ServiceAccount is `cert-manager`** in namespace `cert-manager` — the identity a future exemption should name. - **`optional: true` worked**: the new pod (`plex-c4f87c84d-zgjq5`) is `1/1 Running` with no Secret present, and `/config/certs` was never created. - **The postStart hook RAN and took effect** — the thing the whole hook was written for: `LanNetworksBandwidth="10.0.0.0/8,192.168.0.0/16"` is now set in the live `Preferences.xml`, and `customConnections="https://plex-vip.home.keithjackson.dev:32400"` was applied from the env. The certificate prefs are correctly **ABSENT**, i.e. the hook's "no bundle → leave them alone" guard behaved as designed. - **PiHole's split-horizon answer is correct**, asked of `pihole-0` directly: `plex-vip.home.keithjackson.dev -> 192.168.2.24`, alongside `plex -> .20` and `crafty-mc -> .22`, and the live `/etc/pihole/dnsmasq.conf` carries `address=/plex-vip.home.keithjackson.dev/192.168.2.24`. **This corrects my own earlier check**, which read `.20` and which I nearly reported as a failure — that test resolved via CLUSTER DNS (CoreDNS) from inside a pod and never consulted PiHole at all. - **The gate caught my first attempt and I fixed it rather than working around it.** Keeping the file unregistered produced `MERGE GATE: BLOCKED` (rc 1) with `CHECK FAIL (tls-lifecycle-automated): 1 of 612 assertions failed`. After deleting the file and the dangling EXTRA_UNITS entry: **rc 3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass** — the rc 3 being only the two known environmental skips (`gitea-settings`, `conformance` L2/L3). **Not green. Reported as rc 3.** ## What I did NOT verify - **Whether this actually unblocks the tier.** Unreconciled at time of writing. The reasoning is that pruning the Certificate removes the only failing health check, but I have not seen apps-wave-0 go Ready. - **Whether `http://` restores client connectivity.** PiHole resolves the name and Plex serves 32400 (both measured), but I have not watched a Plex client complete a connection over the advertised URL, and I did not re-check whether plex.tv publishes it. - **THE REMOTE BADGE — the symptom that started this.** `LanNetworksBandwidth` is set, but the log still tags traffic `Allowed Network (WAN)`: **82 WAN, 0 LAN**, with entries continuing after the pref was written. So either Plex needs a restart for it to take effect, or that log tag is not driven by this pref at all. **I do not know which, and the user-visible badge is the only real test** — it is in the Plex client, which I cannot see. - **The Hubble flow export is NOT active.** The cilium agents are still the pre-merge pods, so the `filePath: stdout` question is still open and no flow has been exported. The Cilium rollout — and its LAN-DNS risk, since prod-3 holds the Gateway, PiHole-DNS and Plex VIPs — is still AHEAD. - **Whether a cert-manager exemption is the right shape**, or whether it widens `secret-provenance` more than the operator wants. It is `platform/kyverno/**`, a sensitive path, and deliberately not attempted here. - **Whether the pruned Certificate leaves residue** — I did not check for orphaned `plex-vip-` Secrets or CertificateRequests after the prune. Note Kyverno DENIED every one of those Secrets, so there may be nothing to clean, but that is an inference, not a check. - **Whether `tls-lifecycle-automated` would also have objected to the Certificate living in a `wait: false` Kustomization** (the recommended re-enable shape). It demands a reconciling Flux path, which that provides, but I did not test it. - **The unencrypted advertised URL.** `http://` is tolerable only while remote access does not work (no WAN forward). Nothing enforces that assumption, and nothing will complain if `http` is still there the day the port-forward appears.