fix(plex): unblock the app tier — drop the Certificate that stalled apps-wave-0, advertise http #51

2026-10-04T22:10:57Z

gitea-admin commented

2026-10-04 22:10:57 +00:00

I broke the app tier with #48. This unblocks it. apps-wave-0 has been failing its health check,
so wave-1, wave-2 and everything downstream of platform-config are sitting in
DependencyNotReady. Existing workloads keep running, but no app-tier change can land.

Two faults, both mine, both measured after the merge

1. Kyverno blocks cert-manager in app namespaces. The temporary private-key Secrets were denied:

resource Secret/plex/plex-vip-rp4qj was blocked due to the following policies
secret-provenance:
  secrets-must-be-flux-managed: Secrets in application namespaces must be delivered by Flux
  from SOPS-encrypted git ... or be exempt controller-generated Secrets.

That policy matches every Secret in a namespace labelled
homelab.keithjackson.dev/app-namespace: true, excluding only those labelled
secret-managed-by: controller or carrying a Flux label. plex has that label; gateway — where
the working wildcard cert lives — does not. That asymmetry is why the existing Certificate was
never affected and mine was, and I didn't check it. secretTemplate can't fix it either: those
Secrets come from the keymanager controller, not the Certificate's template. Net result: no
CertificateRequest, Order or Challenge was ever created.

2. Flux health-checks the Certificate object — guarding the pod wasn't enough. optional: true
did exactly its job (the pod rolled and ran fine with no Secret), but apps-wave-0 is wait: true:

health check failed after 5m0.045564329s: timeout waiting for:
  [Certificate/plex/plex-vip status: 'InProgress']

I reasoned about the pod and missed the object. That distinction is the whole bug.

Third, independent: the advertised URL was https:// with no servable certificate — Plex answers
32400 with its *.plex.direct cert, which can't cover plex-vip. That's the exact failure mode of
the outage earlier the same day.

The fix

apps/wave-0/plex-cert.yaml deleted and dropped from resources — pruning it clears the
health gate.
ADVERTISE_IP → http://plex-vip.${BASE_DOMAIN}:32400, which is measured to work today:
PiHole answers the name with 192.168.2.24 (verified against pihole-0) and Plex serves 32400
there. Explicitly temporary; the restore condition is written at the call site.

Deleted, not merely unregistered — my first attempt kept it as "tracked but not onboarded" (the
keycloak/vm-demo shape) and the gate rejected it:

L0 FAIL tls-lifecycle-automated (R-TLS-02)
  apps/wave-0/plex-cert.yaml is not rendered by any Flux Kustomization path — nothing
  reconciles it, so a deleted Certificate/issuer/controller would stay deleted

Legal for a Deployment, not for a TLS object. Good check; I fixed the approach rather than working
around it.

Kept from #48, and verified working

the single-hostname design and PiHole's split-horizon answer
(plex-vip.home.keithjackson.dev -> 192.168.2.24, read from the live dnsmasq.conf)
the postStart hook — LanNetworksBandwidth="10.0.0.0/8,192.168.0.0/16" is now set live, and
the certificate prefs are correctly absent, i.e. the "no bundle → leave them alone" guard worked
the optional schema field, and the TLS env (inert without the Secret, so re-enabling is additive)

To get `https` back — three things, as one change

Exempt cert-manager from secret-provenance (narrowly: an exclude.any[].subjects entry for
ServiceAccount cert-manager/cert-manager). platform/kyverno/** is sensitive — needs a
non-author approval.
Give the Certificate a non-gating home: its own Kustomization with wait: false, the shape
clusters/base/backup.yaml already uses for this reason.
Flip the scheme back.

Recorded in catalog/plex.yaml and decisions/2026-10-04-plex-unblock-app-tier.md.

Gate

rc 3 — spec-immutability, decisions-log, lint, l0, render-golden, l1,
approval-policy all pass; rc 3 is the two known environmental skips. Not green — reported as
rc 3. (First attempt was rc 1 BLOCKED, above.) No sensitive paths touched.

Still open

The Remote badge — the original symptom — is unconfirmed. LanNetworksBandwidth is set, but
the log still tags traffic Allowed Network (WAN) (82 WAN / 0 LAN, continuing after the pref was
written). Either Plex needs a restart for it, or that tag isn't driven by that pref. The badge in
your client is the only real test.
The Hubble export from #50 is not active yet — the cilium agents are still pre-merge pods, so
filePath: stdout is still unproven and the Cilium rollout (with its LAN-DNS risk, since prod-3
holds the Gateway, PiHole-DNS and Plex VIPs) is still ahead.

🤖 Generated with Claude Code

**I broke the app tier with #48. This unblocks it.** `apps-wave-0` has been failing its health check, so wave-1, wave-2 and everything downstream of `platform-config` are sitting in `DependencyNotReady`. Existing workloads keep running, but no app-tier change can land. ## Two faults, both mine, both measured after the merge **1. Kyverno blocks cert-manager in app namespaces.** The temporary private-key Secrets were denied: ``` resource Secret/plex/plex-vip-rp4qj was blocked due to the following policies secret-provenance: secrets-must-be-flux-managed: Secrets in application namespaces must be delivered by Flux from SOPS-encrypted git ... or be exempt controller-generated Secrets. ``` That policy matches every Secret in a namespace labelled `homelab.keithjackson.dev/app-namespace: true`, excluding only those labelled `secret-managed-by: controller` or carrying a Flux label. **`plex` has that label; `gateway` — where the working wildcard cert lives — does not.** That asymmetry is why the existing Certificate was never affected and mine was, and I didn't check it. `secretTemplate` can't fix it either: those Secrets come from the keymanager controller, not the Certificate's template. Net result: **no CertificateRequest, Order or Challenge was ever created.** **2. Flux health-checks the Certificate object — guarding the pod wasn't enough.** `optional: true` did exactly its job (the pod rolled and ran fine with no Secret), but `apps-wave-0` is `wait: true`: ``` health check failed after 5m0.045564329s: timeout waiting for: [Certificate/plex/plex-vip status: 'InProgress'] ``` I reasoned about the pod and missed the object. That distinction is the whole bug. **Third, independent:** the advertised URL was `https://` with no servable certificate — Plex answers 32400 with its `*.plex.direct` cert, which can't cover `plex-vip`. That's the exact failure mode of the outage earlier the same day. ## The fix - **`apps/wave-0/plex-cert.yaml` deleted** and dropped from `resources` — pruning it clears the health gate. - **`ADVERTISE_IP` → `http://plex-vip.${BASE_DOMAIN}:32400`**, which is *measured* to work today: PiHole answers the name with `192.168.2.24` (verified against `pihole-0`) and Plex serves 32400 there. Explicitly temporary; the restore condition is written at the call site. **Deleted, not merely unregistered** — my first attempt kept it as "tracked but not onboarded" (the `keycloak`/`vm-demo` shape) and **the gate rejected it**: ``` L0 FAIL tls-lifecycle-automated (R-TLS-02) apps/wave-0/plex-cert.yaml is not rendered by any Flux Kustomization path — nothing reconciles it, so a deleted Certificate/issuer/controller would stay deleted ``` Legal for a Deployment, not for a TLS object. Good check; I fixed the approach rather than working around it. ## Kept from #48, and verified working - the single-hostname design and PiHole's split-horizon answer (`plex-vip.home.keithjackson.dev -> 192.168.2.24`, read from the live `dnsmasq.conf`) - the `postStart` hook — **`LanNetworksBandwidth="10.0.0.0/8,192.168.0.0/16"` is now set live**, and the certificate prefs are correctly absent, i.e. the "no bundle → leave them alone" guard worked - the `optional` schema field, and the TLS env (inert without the Secret, so re-enabling is additive) ## To get `https` back — three things, as one change 1. Exempt cert-manager from `secret-provenance` (narrowly: an `exclude.any[].subjects` entry for ServiceAccount `cert-manager`/`cert-manager`). `platform/kyverno/**` is sensitive — needs a non-author approval. 2. Give the Certificate a **non-gating** home: its own Kustomization with `wait: false`, the shape `clusters/base/backup.yaml` already uses for this reason. 3. Flip the scheme back. Recorded in `catalog/plex.yaml` and `decisions/2026-10-04-plex-unblock-app-tier.md`. ## Gate **rc 3** — `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1`, `approval-policy` all pass; rc 3 is the two known environmental skips. **Not green — reported as rc 3.** (First attempt was rc 1 BLOCKED, above.) No sensitive paths touched. ## Still open - **The Remote badge — the original symptom — is unconfirmed.** `LanNetworksBandwidth` is set, but the log still tags traffic `Allowed Network (WAN)` (82 WAN / 0 LAN, continuing after the pref was written). Either Plex needs a restart for it, or that tag isn't driven by that pref. The badge in your client is the only real test. - **The Hubble export from #50 is not active yet** — the cilium agents are still pre-merge pods, so `filePath: stdout` is still unproven and the Cilium rollout (with its LAN-DNS risk, since prod-3 holds the Gateway, PiHole-DNS and Plex VIPs) is still ahead. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea-admin added 1 commit 2026-10-04 22:10:57 +00:00

fix(plex): unblock the app tier — drop the Certificate that stalled apps-wave-0, advertise http

gate / gate (pull_request) Successful in 1m50s

Details

gate / conformance (pull_request) Has been skipped

Details

a0e6ef145f

I broke the app tier with #48. Two faults, both mine, both measured after merge:

1. KYVERNO BLOCKS cert-manager IN APP NAMESPACES. The temporary private-key
   Secrets (plex-vip-<random>) were denied by
   secret-provenance/secrets-must-be-flux-managed, which matches every Secret in
   a namespace labelled homelab.keithjackson.dev/app-namespace=true and excludes
   only those labelled secret-managed-by=controller or carrying a Flux label.
   `plex` HAS that label; `gateway` -- where the working wildcard cert lives --
   does NOT. That asymmetry is why the existing Certificate was never affected
   and mine was, and I did not check it. secretTemplate cannot fix it: those
   Secrets come from the keymanager controller, not the Certificate template. So
   no CertificateRequest, Order or Challenge was ever created.

2. FLUX HEALTH-CHECKS THE CERTIFICATE. `optional: true` did its job -- the pod
   rolled and ran fine without the Secret -- but apps-wave-0 is wait:true, so
   Flux waits on the Certificate OBJECT:
     timeout waiting for: [Certificate/plex/plex-vip status: 'InProgress']
   apps-wave-0 never went Ready; wave-1/-2 and everything downstream of
   platform-config sat in DependencyNotReady behind it. I guarded the pod and
   missed the object. That distinction is the whole bug.

Third, independent: the advertised URL was https with no servable certificate --
Plex answers 32400 with its *.plex.direct cert, which cannot cover plex-vip --
the exact failure mode of the outage earlier the same day.

So: delete the Certificate (pruning it clears the health gate) and advertise
http://plex-vip:32400, which is MEASURED to work today -- PiHole answers the
name with the Plex VIP (verified against pihole-0) and Plex serves 32400 there.
http is explicitly temporary and only tolerable because remote access does not
work yet; the restore condition is written at the call site.

DELETED, not merely unregistered: my first attempt kept the file as "tracked but
not onboarded" and the gate rejected it -- tls-lifecycle-automated (R-TLS-02)
fails any Certificate no Flux path reconciles. Legal for a Deployment, not for a
TLS object. The re-enable recipe (Kyverno exemption + a wait:false Kustomization
+ the https flip, as ONE change) is recorded in catalog/plex.yaml.

Kept and verified working from #48: the hostname design, PiHole's split-horizon
answer, and the postStart hook -- LanNetworksBandwidth is now set live.

Gate: rc 3, all substantive stages pass (first attempt was rc 1 BLOCKED).

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

jakman85 merged commit bc7095677b into main

2026-10-05 13:53:21 +00:00

jakman85 referenced this issue from a commit

2026-10-05 13:53:22 +00:00

Merge pull request 'fix(plex): unblock the app tier — drop the Certificate that stalled apps-wave-0, advertise http' (#51) from fix/plex-unblock-app-tier into main

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Home-Network/home-network-k3s-impl#51