jakman85andClaude Opus 5
a0e6ef145f
I broke the app tier with #48. Two faults, both mine, both measured after merge: 1. KYVERNO BLOCKS cert-manager IN APP NAMESPACES. The temporary private-key Secrets (plex-vip-<random>) were denied by secret-provenance/secrets-must-be-flux-managed, which matches every Secret in a namespace labelled homelab.keithjackson.dev/app-namespace=true and excludes only those labelled secret-managed-by=controller or carrying a Flux label. `plex` HAS that label; `gateway` -- where the working wildcard cert lives -- does NOT. That asymmetry is why the existing Certificate was never affected and mine was, and I did not check it. secretTemplate cannot fix it: those Secrets come from the keymanager controller, not the Certificate template. So no CertificateRequest, Order or Challenge was ever created. 2. FLUX HEALTH-CHECKS THE CERTIFICATE. `optional: true` did its job -- the pod rolled and ran fine without the Secret -- but apps-wave-0 is wait:true, so Flux waits on the Certificate OBJECT: timeout waiting for: [Certificate/plex/plex-vip status: 'InProgress'] apps-wave-0 never went Ready; wave-1/-2 and everything downstream of platform-config sat in DependencyNotReady behind it. I guarded the pod and missed the object. That distinction is the whole bug. Third, independent: the advertised URL was https with no servable certificate -- Plex answers 32400 with its *.plex.direct cert, which cannot cover plex-vip -- the exact failure mode of the outage earlier the same day. So: delete the Certificate (pruning it clears the health gate) and advertise http://plex-vip:32400, which is MEASURED to work today -- PiHole answers the name with the Plex VIP (verified against pihole-0) and Plex serves 32400 there. http is explicitly temporary and only tolerable because remote access does not work yet; the restore condition is written at the call site. DELETED, not merely unregistered: my first attempt kept the file as "tracked but not onboarded" and the gate rejected it -- tls-lifecycle-automated (R-TLS-02) fails any Certificate no Flux path reconciles. Legal for a Deployment, not for a TLS object. The re-enable recipe (Kyverno exemption + a wait:false Kustomization + the https flip, as ONE change) is recorded in catalog/plex.yaml. Kept and verified working from #48: the hostname design, PiHole's split-horizon answer, and the postStart hook -- LanNetworksBandwidth is now set live. Gate: rc 3, all substantive stages pass (first attempt was rc 1 BLOCKED). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
127 lines
8.6 KiB
Markdown
127 lines
8.6 KiB
Markdown
---
|
|
date: 2026-10-04
|
|
branch: fix/plex-unblock-app-tier
|
|
title: Unblock the app tier — my Certificate stalled apps-wave-0, and the advertised https URL had no servable certificate
|
|
---
|
|
|
|
## What changed
|
|
|
|
**I broke the app tier with PR #48 and this reverts the part that did it.** Two faults, both mine,
|
|
both measured after the merge:
|
|
|
|
1. **Kyverno blocks cert-manager in app namespaces.** The temporary private-key Secrets
|
|
(`plex-vip-<random>`) were denied by `secret-provenance/secrets-must-be-flux-managed`:
|
|
|
|
resource Secret/plex/plex-vip-rp4qj was blocked due to the following policies
|
|
secret-provenance:
|
|
secrets-must-be-flux-managed: Secrets in application namespaces must be delivered by
|
|
Flux from SOPS-encrypted git ... or be exempt controller-generated Secrets.
|
|
|
|
The policy matches every Secret in a namespace labelled
|
|
`homelab.keithjackson.dev/app-namespace: true` and excludes only Secrets labelled
|
|
`homelab.keithjackson.dev/secret-managed-by: controller` or carrying a Flux label. **The `plex`
|
|
namespace has that label; the `gateway` namespace — where the working wildcard cert lives — does
|
|
NOT.** That asymmetry is why the existing Certificate was never affected and mine was, and I did
|
|
not check it. cert-manager's temp key Secrets carry neither exempting label, and `secretTemplate`
|
|
cannot label them (the keymanager controller creates them, not the Certificate template). Result:
|
|
**no CertificateRequest, Order or Challenge was ever created.**
|
|
|
|
2. **Flux health-checks the Certificate — guarding the pod was not enough.** `optional: true` did
|
|
exactly what it was designed to do: Plex rolled and ran fine without the Secret (verified). But
|
|
`apps-wave-0` is `wait: true`, so Flux waits on every applied object with a known health type:
|
|
|
|
health check failed after 5m0.045564329s: timeout waiting for:
|
|
[Certificate/plex/plex-vip status: 'InProgress']
|
|
|
|
apps-wave-0 never went Ready, and apps-wave-1/-2 plus everything downstream of platform-config
|
|
sat in `DependencyNotReady` behind it. I reasoned about the pod and missed the object.
|
|
|
|
A third, independent problem the same PR introduced: **the advertised URL was `https://` with no
|
|
servable certificate.** Plex answers 32400 with its `*.<serverhash>.plex.direct` cert, which cannot
|
|
cover `plex-vip.${BASE_DOMAIN}` — the precise failure mode of the outage earlier that day.
|
|
|
|
- **`apps/wave-0/plex-cert.yaml` DELETED**, and `plex-cert.yaml` removed from
|
|
`apps/wave-0/kustomization.yaml` `resources`, so Flux prunes the stuck Certificate and the health
|
|
gate clears. **Deleted rather than merely unregistered** — my first attempt kept the file as
|
|
"tracked but not onboarded" (the shape `keycloak.yaml` and `vm-demo.yaml` use) and **the repo's own
|
|
gate rejected it**: `tls-lifecycle-automated` (R-TLS-02) fails any Certificate in the tree that no
|
|
Flux path reconciles — *"nothing reconciles it, so a deleted Certificate/issuer/controller would
|
|
stay deleted until a human re-applied it."* That pattern is legal for a Deployment and not for a
|
|
TLS object, which is a distinction I had not known.
|
|
- **`catalog/render.py`:** the now-dangling `plex-cert` EXTRA_UNITS entry removed.
|
|
- **The re-enable recipe is recorded at the call site** — in `catalog/plex.yaml` beside the scheme
|
|
it is coupled to — rather than in the deleted file: the Kyverno exemption, the non-gating
|
|
Kustomization, and the `https` flip, as one change.
|
|
- **`catalog/plex.yaml`:** `ADVERTISE_IP` → **`http://`**`plex-vip.${BASE_DOMAIN}:32400`, explicitly
|
|
temporary, with the restore condition written at the call site.
|
|
|
|
**Everything else from PR #48 is kept and working** — the hostname design, the PiHole split-horizon
|
|
answer, the `LanNetworksBandwidth` hook, the `optional` schema field, and the TLS env (inert without
|
|
the Secret, so re-enabling is additive).
|
|
|
|
## What I verified
|
|
|
|
Live prod, 2026-10-04, read-only, after the #48/#50 merges.
|
|
|
|
- **The stall and its exact cause**: the health-check timeout message above, and the full dependency
|
|
cascade (`cilium` → "dependency 'flux-system/flux-system' revision is not up to date", everything
|
|
else `DependencyNotReady`). The root `cilium` wait was a normal revision-gated cascade; the
|
|
*Certificate* was the real blocker.
|
|
- **The Kyverno denial text, verbatim**, from the cert-manager controller log, repeating every ~30s
|
|
across at least 15 distinct Secret names.
|
|
- **The policy's own structure**, read from `platform/kyverno/admission/secret-provenance.yaml`:
|
|
match = Secrets in `app-namespace: true` namespaces; exclude = `secret-managed-by: controller` OR
|
|
a `kustomize.toolkit.fluxcd.io/name` label; validate = unconditional `deny`.
|
|
- **The namespace-label asymmetry**: `plex` carries
|
|
`homelab.keithjackson.dev/app-namespace: "true"`; `gateway` does not.
|
|
- **cert-manager's controller ServiceAccount is `cert-manager`** in namespace `cert-manager` — the
|
|
identity a future exemption should name.
|
|
- **`optional: true` worked**: the new pod (`plex-c4f87c84d-zgjq5`) is `1/1 Running` with no Secret
|
|
present, and `/config/certs` was never created.
|
|
- **The postStart hook RAN and took effect** — the thing the whole hook was written for:
|
|
`LanNetworksBandwidth="10.0.0.0/8,192.168.0.0/16"` is now set in the live `Preferences.xml`, and
|
|
`customConnections="https://plex-vip.home.keithjackson.dev:32400"` was applied from the env. The
|
|
certificate prefs are correctly **ABSENT**, i.e. the hook's "no bundle → leave them alone" guard
|
|
behaved as designed.
|
|
- **PiHole's split-horizon answer is correct**, asked of `pihole-0` directly:
|
|
`plex-vip.home.keithjackson.dev -> 192.168.2.24`, alongside `plex -> .20` and `crafty-mc -> .22`,
|
|
and the live `/etc/pihole/dnsmasq.conf` carries
|
|
`address=/plex-vip.home.keithjackson.dev/192.168.2.24`. **This corrects my own earlier check**,
|
|
which read `.20` and which I nearly reported as a failure — that test resolved via CLUSTER DNS
|
|
(CoreDNS) from inside a pod and never consulted PiHole at all.
|
|
- **The gate caught my first attempt and I fixed it rather than working around it.** Keeping the
|
|
file unregistered produced `MERGE GATE: BLOCKED` (rc 1) with
|
|
`CHECK FAIL (tls-lifecycle-automated): 1 of 612 assertions failed`. After deleting the file and
|
|
the dangling EXTRA_UNITS entry: **rc 3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`,
|
|
`render-golden`, `l1` and `approval-policy` all **pass** — the rc 3 being only the two known
|
|
environmental skips (`gitea-settings`, `conformance` L2/L3). **Not green. Reported as rc 3.**
|
|
|
|
## What I did NOT verify
|
|
|
|
- **Whether this actually unblocks the tier.** Unreconciled at time of writing. The reasoning is
|
|
that pruning the Certificate removes the only failing health check, but I have not seen
|
|
apps-wave-0 go Ready.
|
|
- **Whether `http://` restores client connectivity.** PiHole resolves the name and Plex serves 32400
|
|
(both measured), but I have not watched a Plex client complete a connection over the advertised
|
|
URL, and I did not re-check whether plex.tv publishes it.
|
|
- **THE REMOTE BADGE — the symptom that started this.** `LanNetworksBandwidth` is set, but the log
|
|
still tags traffic `Allowed Network (WAN)`: **82 WAN, 0 LAN**, with entries continuing after the
|
|
pref was written. So either Plex needs a restart for it to take effect, or that log tag is not
|
|
driven by this pref at all. **I do not know which, and the user-visible badge is the only real
|
|
test** — it is in the Plex client, which I cannot see.
|
|
- **The Hubble flow export is NOT active.** The cilium agents are still the pre-merge pods, so the
|
|
`filePath: stdout` question is still open and no flow has been exported. The Cilium rollout — and
|
|
its LAN-DNS risk, since prod-3 holds the Gateway, PiHole-DNS and Plex VIPs — is still AHEAD.
|
|
- **Whether a cert-manager exemption is the right shape**, or whether it widens
|
|
`secret-provenance` more than the operator wants. It is `platform/kyverno/**`, a sensitive path,
|
|
and deliberately not attempted here.
|
|
- **Whether the pruned Certificate leaves residue** — I did not check for orphaned
|
|
`plex-vip-<random>` Secrets or CertificateRequests after the prune. Note Kyverno DENIED every one
|
|
of those Secrets, so there may be nothing to clean, but that is an inference, not a check.
|
|
- **Whether `tls-lifecycle-automated` would also have objected to the Certificate living in a
|
|
`wait: false` Kustomization** (the recommended re-enable shape). It demands a reconciling Flux
|
|
path, which that provides, but I did not test it.
|
|
- **The unencrypted advertised URL.** `http://` is tolerable only while remote access does not work
|
|
(no WAN forward). Nothing enforces that assumption, and nothing will complain if `http` is still
|
|
there the day the port-forward appears.
|