home-network-k3s-impl/decisions/2026-10-04-plex-unblock-app-tier.md

date, branch, title

date	branch	title
2026-10-04	fix/plex-unblock-app-tier	Unblock the app tier — my Certificate stalled apps-wave-0, and the advertised https URL had no servable certificate

What changed

I broke the app tier with PR #48 and this reverts the part that did it. Two faults, both mine, both measured after the merge:

1. Kyverno blocks cert-manager in app namespaces. The temporary private-key Secrets (plex-vip-<random>) were denied by secret-provenance/secrets-must-be-flux-managed:

   resource Secret/plex/plex-vip-rp4qj was blocked due to the following policies
   secret-provenance:
     secrets-must-be-flux-managed: Secrets in application namespaces must be delivered by
     Flux from SOPS-encrypted git ... or be exempt controller-generated Secrets.
   

   The policy matches every Secret in a namespace labelled homelab.keithjackson.dev/app-namespace: true and excludes only Secrets labelled homelab.keithjackson.dev/secret-managed-by: controller or carrying a Flux label. The plex namespace has that label; the gateway namespace — where the working wildcard cert lives — does NOT. That asymmetry is why the existing Certificate was never affected and mine was, and I did not check it. cert-manager's temp key Secrets carry neither exempting label, and secretTemplate cannot label them (the keymanager controller creates them, not the Certificate template). Result: no CertificateRequest, Order or Challenge was ever created.

2. Flux health-checks the Certificate — guarding the pod was not enough. optional: true did exactly what it was designed to do: Plex rolled and ran fine without the Secret (verified). But apps-wave-0 is wait: true, so Flux waits on every applied object with a known health type:

   health check failed after 5m0.045564329s: timeout waiting for:
   [Certificate/plex/plex-vip status: 'InProgress']
   

   apps-wave-0 never went Ready, and apps-wave-1/-2 plus everything downstream of platform-config sat in DependencyNotReady behind it. I reasoned about the pod and missed the object.

A third, independent problem the same PR introduced: the advertised URL was https:// with no servable certificate. Plex answers 32400 with its *.<serverhash>.plex.direct cert, which cannot cover plex-vip.${BASE_DOMAIN} — the precise failure mode of the outage earlier that day.

• apps/wave-0/plex-cert.yaml DELETED, and plex-cert.yaml removed from apps/wave-0/kustomization.yaml resources, so Flux prunes the stuck Certificate and the health gate clears. Deleted rather than merely unregistered — my first attempt kept the file as "tracked but not onboarded" (the shape keycloak.yaml and vm-demo.yaml use) and the repo's own gate rejected it: tls-lifecycle-automated (R-TLS-02) fails any Certificate in the tree that no Flux path reconciles — "nothing reconciles it, so a deleted Certificate/issuer/controller would stay deleted until a human re-applied it." That pattern is legal for a Deployment and not for a TLS object, which is a distinction I had not known.
• catalog/render.py: the now-dangling plex-cert EXTRA_UNITS entry removed.
• The re-enable recipe is recorded at the call site — in catalog/plex.yaml beside the scheme it is coupled to — rather than in the deleted file: the Kyverno exemption, the non-gating Kustomization, and the https flip, as one change.
• catalog/plex.yaml: ADVERTISE_IP → http://plex-vip.${BASE_DOMAIN}:32400, explicitly temporary, with the restore condition written at the call site.

Everything else from PR #48 is kept and working — the hostname design, the PiHole split-horizon answer, the LanNetworksBandwidth hook, the optional schema field, and the TLS env (inert without the Secret, so re-enabling is additive).

What I verified

Live prod, 2026-10-04, read-only, after the #48/#50 merges.

• The stall and its exact cause: the health-check timeout message above, and the full dependency cascade (cilium → "dependency 'flux-system/flux-system' revision is not up to date", everything else DependencyNotReady). The root cilium wait was a normal revision-gated cascade; the Certificate was the real blocker.
• The Kyverno denial text, verbatim, from the cert-manager controller log, repeating every ~30s across at least 15 distinct Secret names.
• The policy's own structure, read from platform/kyverno/admission/secret-provenance.yaml: match = Secrets in app-namespace: true namespaces; exclude = secret-managed-by: controller OR a kustomize.toolkit.fluxcd.io/name label; validate = unconditional deny.
• The namespace-label asymmetry: plex carries homelab.keithjackson.dev/app-namespace: "true"; gateway does not.
• cert-manager's controller ServiceAccount is cert-manager in namespace cert-manager — the identity a future exemption should name.
• optional: true worked: the new pod (plex-c4f87c84d-zgjq5) is 1/1 Running with no Secret present, and /config/certs was never created.
• The postStart hook RAN and took effect — the thing the whole hook was written for: LanNetworksBandwidth="10.0.0.0/8,192.168.0.0/16" is now set in the live Preferences.xml, and customConnections="https://plex-vip.home.keithjackson.dev:32400" was applied from the env. The certificate prefs are correctly ABSENT, i.e. the hook's "no bundle → leave them alone" guard behaved as designed.
• PiHole's split-horizon answer is correct, asked of pihole-0 directly: plex-vip.home.keithjackson.dev -> 192.168.2.24, alongside plex -> .20 and crafty-mc -> .22, and the live /etc/pihole/dnsmasq.conf carries address=/plex-vip.home.keithjackson.dev/192.168.2.24. This corrects my own earlier check, which read .20 and which I nearly reported as a failure — that test resolved via CLUSTER DNS (CoreDNS) from inside a pod and never consulted PiHole at all.
• The gate caught my first attempt and I fixed it rather than working around it. Keeping the file unregistered produced MERGE GATE: BLOCKED (rc 1) with CHECK FAIL (tls-lifecycle-automated): 1 of 612 assertions failed. After deleting the file and the dangling EXTRA_UNITS entry: rc 3, with spec-immutability, decisions-log, lint, l0, render-golden, l1 and approval-policy all pass — the rc 3 being only the two known environmental skips (gitea-settings, conformance L2/L3). Not green. Reported as rc 3.

What I did NOT verify

• Whether this actually unblocks the tier. Unreconciled at time of writing. The reasoning is that pruning the Certificate removes the only failing health check, but I have not seen apps-wave-0 go Ready.
• Whether http:// restores client connectivity. PiHole resolves the name and Plex serves 32400 (both measured), but I have not watched a Plex client complete a connection over the advertised URL, and I did not re-check whether plex.tv publishes it.
• THE REMOTE BADGE — the symptom that started this. LanNetworksBandwidth is set, but the log still tags traffic Allowed Network (WAN): 82 WAN, 0 LAN, with entries continuing after the pref was written. So either Plex needs a restart for it to take effect, or that log tag is not driven by this pref at all. I do not know which, and the user-visible badge is the only real test — it is in the Plex client, which I cannot see.
• The Hubble flow export is NOT active. The cilium agents are still the pre-merge pods, so the filePath: stdout question is still open and no flow has been exported. The Cilium rollout — and its LAN-DNS risk, since prod-3 holds the Gateway, PiHole-DNS and Plex VIPs — is still AHEAD.
• Whether a cert-manager exemption is the right shape, or whether it widens secret-provenance more than the operator wants. It is platform/kyverno/**, a sensitive path, and deliberately not attempted here.
• Whether the pruned Certificate leaves residue — I did not check for orphaned plex-vip-<random> Secrets or CertificateRequests after the prune. Note Kyverno DENIED every one of those Secrets, so there may be nothing to clean, but that is an inference, not a check.
• Whether tls-lifecycle-automated would also have objected to the Certificate living in a wait: false Kustomization (the recommended re-enable shape). It demands a reconciling Flux path, which that provides, but I did not test it.
• The unencrypted advertised URL. http:// is tolerable only while remote access does not work (no WAN forward). Nothing enforces that assumption, and nothing will complain if http is still there the day the port-forward appears.