fix(plex): unblock the app tier — drop the Certificate that stalled apps-wave-0, advertise http #51
No files matched your search
@@ -18,7 +18,6 @@ resources:
|
||||
- pihole-dns.yaml
|
||||
- pihole-oauth2-proxy-credentials.sops.yaml
|
||||
- pihole.yaml
|
||||
- plex-cert.yaml
|
||||
- plex-claim-secret.sops.yaml
|
||||
- plex-remote.yaml
|
||||
- plex.yaml
|
||||
|
||||
@@ -1,66 +0,0 @@
|
||||
---
|
||||
# =================================================================================================
|
||||
# PLEX TLS — a Let's Encrypt certificate for the ONE hostname Plex advertises to plex.tv.
|
||||
# =================================================================================================
|
||||
# HAND-WRITTEN, NOT RENDERED. A non-catalog app-tier unit anchored to the `plex` ServiceDefinition
|
||||
# and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS table — the same
|
||||
# mechanism gitea-actions, pihole-dns and plex-remote use. It is hand-written for the same reason:
|
||||
# `schemas/servicedefinition.schema.json` is `additionalProperties: false` and has NO field for a
|
||||
# cert-manager Certificate. (`ingressRoutes` gets TLS for free from the Gateway's wildcard; this is
|
||||
# the opposite case — a certificate for a workload that terminates its OWN TLS on a raw L4 port.)
|
||||
#
|
||||
# WHY PLEX NEEDS ITS OWN CERTIFICATE AT ALL. Plex serves tcp/32400 directly (not through the L7
|
||||
# Gateway — the operator chose the native path 2026-10-03, apps/wave-0/plex-remote.yaml) and it
|
||||
# presents its own `*.<serverhash>.plex.direct` certificate there. That name cannot cover
|
||||
# `plex-vip.${BASE_DOMAIN}`, the hostname we advertise, so without this the advertised `https://`
|
||||
# URL fails hostname verification for every client. That is not hypothetical: it is the shape of
|
||||
# the 2026-10-04 outage, where Plex's log carried a continuous storm of
|
||||
# `CERT: incomplete TLS handshake ... sslv3 alert certificate unknown` from real LAN clients while
|
||||
# plex.tv could publish nothing and the server vanished from the account.
|
||||
#
|
||||
# A SAN match is not what makes Plex FUNCTION — `httpsRequired=0` and `secureConnections` default
|
||||
# to Preferred, so clients can fall back to cleartext. It is what keeps the REMOTE leg encrypted
|
||||
# rather than silently downgrading traffic that crosses the internet.
|
||||
#
|
||||
# HOW IT REACHES PLEX. catalog/plex.yaml takes `tls.crt`/`tls.key` from this Secret as env
|
||||
# (`optional: true` — see below), and its `lifecycle.postStart` hook converts the PEM into the
|
||||
# PKCS#12 bundle Plex's `customCertificatePath` requires and sets the three `customCertificate*`
|
||||
# prefs. The conversion is redone on EVERY pod start, so a cert-manager RENEWAL is picked up with
|
||||
# no extra moving part and nothing to remember.
|
||||
#
|
||||
# ORDERING, AND WHY THE ENV REFS ARE OPTIONAL. This Certificate must live in the `plex` namespace
|
||||
# for the Deployment to read its Secret, and that namespace is created by the rendered
|
||||
# apps/wave-0/plex.yaml ITSELF — so this cannot be issued in an earlier tier (platform-config runs
|
||||
# before the namespace exists) and lands in the SAME reconcile as the workload. On a cold bootstrap
|
||||
# the Secret is therefore absent for as long as a DNS01 challenge takes. With hard env refs Plex
|
||||
# would sit in CreateContainerConfigError until then, and because `apps-wave-0` is `wait: true`
|
||||
# with `apps-wave-1` dependsOn it, a failed or slow issuance would STALL THE WHOLE APP TIER. Hence
|
||||
# `optional: true` on both refs (schema 1.11.0, added for exactly this): no certificate means Plex
|
||||
# still starts and serves on its own plex.direct cert, the hook logs that TLS was not installed,
|
||||
# and the next start after issuance converges.
|
||||
#
|
||||
# ISSUER: ClusterIssuer `letsencrypt-prod` (platform/cert-manager/clusterissuers.yaml), the same
|
||||
# issuer behind the Gateway's wildcard (platform/gateway/wildcard-certificate.yaml, Ready 28d on
|
||||
# prod as measured 2026-10-04). A SINGLE name is requested rather than reusing the wildcard,
|
||||
# because the wildcard's Secret lives in the `gateway` namespace and a Secret cannot be read across
|
||||
# namespaces; requesting `*.${BASE_DOMAIN}` again here would also burn a duplicate-certificate slot
|
||||
# against Let's Encrypt's weekly limit for no benefit. A single name is the least-privilege shape:
|
||||
# this key signs for `plex-vip.${BASE_DOMAIN}` and nothing else.
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: plex-vip
|
||||
namespace: plex
|
||||
labels:
|
||||
app.kubernetes.io/name: plex
|
||||
spec:
|
||||
# The name Plex ADVERTISES and the name on this certificate must stay identical. They are two
|
||||
# halves of one decision: catalog/plex.yaml's ADVERTISE_IP, the postStart hook's
|
||||
# customCertificateDomain, this dnsNames entry, and catalog/pihole.yaml's split-horizon answer.
|
||||
# Nothing enforces that they agree.
|
||||
dnsNames:
|
||||
- plex-vip.${BASE_DOMAIN}
|
||||
secretName: plex-vip-tls
|
||||
issuerRef:
|
||||
kind: ClusterIssuer
|
||||
name: letsencrypt-prod
|
||||
@@ -138,7 +138,7 @@ spec:
|
||||
name: plex-claim
|
||||
key: PLEX_CLAIM
|
||||
- name: ADVERTISE_IP
|
||||
value: https://plex-vip.${BASE_DOMAIN}:32400
|
||||
value: http://plex-vip.${BASE_DOMAIN}:32400
|
||||
- name: PLEX_TLS_CRT
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
|
||||
+35
-1
@@ -107,8 +107,42 @@ env:
|
||||
# OVERLAY fs rather than the /config PVC — so this is re-applied on EVERY pod start and git, not
|
||||
# a web form, owns the value. Hand-typed custom URLs are what broke this service; they are now
|
||||
# overwritten on every start.
|
||||
#
|
||||
# SCHEME IS `http` AND THAT IS TEMPORARY, 2026-10-04. It was `https`, which requires Plex to serve
|
||||
# a certificate for this name — and the Certificate that would have supplied one is NOT APPLIED
|
||||
# (see apps/wave-0/plex-cert.yaml's header). Advertising `https` without it reproduces the
|
||||
# 2026-10-04 outage exactly: Plex answers 32400 with its `*.<serverhash>.plex.direct` cert, every
|
||||
# TLS-validating client rejects the hostname, and plex.tv can publish nothing. `http` works TODAY
|
||||
# and is MEASURED to: PiHole answers plex-vip.${BASE_DOMAIN} with ${PLEX_L4_VIP} (verified live
|
||||
# against pihole-0, 2026-10-04) and Plex serves 32400 there.
|
||||
#
|
||||
# THE COST, STATED: the remote leg is unencrypted. That is tolerable only because remote access
|
||||
# does not work yet at all (the WAN tcp/32400 forward is still absent), so today this carries LAN
|
||||
# traffic only. RESTORE `https` IN THE SAME COMMIT that re-applies the Certificate — the two are
|
||||
# one change, and splitting them is what broke this.
|
||||
#
|
||||
# HOW TO GET `https` BACK. The Certificate that would serve this name was applied on 2026-10-04
|
||||
# and STALLED THE ENTIRE APP TIER; the file was DELETED rather than merely unregistered, because
|
||||
# `tls-lifecycle-automated` (R-TLS-02) fails any Certificate left in the tree that no Flux path
|
||||
# reconciles. Full post-mortem: decisions/2026-10-04-plex-unblock-app-tier.md. Three things must
|
||||
# land TOGETHER:
|
||||
# 1. EXEMPT cert-manager FROM `secret-provenance`. Its temporary private-key Secrets
|
||||
# (`plex-vip-<random>`) are denied by `secrets-must-be-flux-managed`, which matches every
|
||||
# Secret in a namespace labelled `homelab.keithjackson.dev/app-namespace: true` — `plex` HAS
|
||||
# that label, `gateway` (where the working wildcard cert lives) does NOT, which is why this
|
||||
# never bit the existing Certificate. `secretTemplate` cannot fix it: those Secrets come from
|
||||
# the keymanager controller, not the Certificate's template. Narrow fix: an
|
||||
# `exclude.any[].subjects` entry for ServiceAccount `cert-manager`/`cert-manager` (confirmed
|
||||
# live as the controller's SA). `platform/kyverno/**` is a SENSITIVE path — non-author approval.
|
||||
# 2. GIVE THE CERTIFICATE A NON-GATING HOME. `apps-wave-0` is `wait: true`, so Flux health-checks
|
||||
# the Certificate OBJECT and a pending issuance stalls the whole tier
|
||||
# (`timeout waiting for: [Certificate/plex/plex-vip status: 'InProgress']`). `optional: true`
|
||||
# on the env refs below protects the POD, not the Kustomization — that distinction is the
|
||||
# whole bug. Use a separate Kustomization with `wait: false`, the shape
|
||||
# clusters/base/backup.yaml already uses for exactly this reason.
|
||||
# 3. Flip this value back to `https`.
|
||||
- name: ADVERTISE_IP
|
||||
value: https://plex-vip.${BASE_DOMAIN}:32400
|
||||
value: http://plex-vip.${BASE_DOMAIN}:32400
|
||||
# THE TLS MATERIAL FOR THE NAME ABOVE, as PEM text in env rather than a mounted volume.
|
||||
# `plex-vip-tls` is issued by cert-manager from ClusterIssuer `letsencrypt-prod` for exactly
|
||||
# `plex-vip.${BASE_DOMAIN}` (apps/wave-0/plex-cert.yaml, same wave => same reconcile as this
|
||||
|
||||
@@ -2209,23 +2209,6 @@ EXTRA_UNITS: dict = {
|
||||
"plex-remote.yaml",
|
||||
],
|
||||
},
|
||||
# Plex's own TLS certificate for the single hostname it advertises to plex.tv
|
||||
# (apps/wave-0/plex-cert.yaml). `above: False` co-locates it in plex's OWN wave, which is the
|
||||
# only placement available: the Certificate must land in the `plex` namespace, and that
|
||||
# namespace is created by the rendered plex.yaml in this very wave, so it CANNOT be issued
|
||||
# earlier. The workload tolerates the resulting gap by marking the env refs `optional` rather
|
||||
# than by ordering -- see the header of plex-cert.yaml.
|
||||
#
|
||||
# Non-catalog because the ServiceDefinition schema has no field for a cert-manager Certificate
|
||||
# (additionalProperties is false); `ingressRoutes` inherits the Gateway wildcard instead, which
|
||||
# is the opposite case to a workload terminating its own TLS on a raw L4 port.
|
||||
"plex-cert": {
|
||||
"anchor": "plex",
|
||||
"above": False,
|
||||
"files": [
|
||||
"plex-cert.yaml",
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,126 @@
|
||||
---
|
||||
date: 2026-10-04
|
||||
branch: fix/plex-unblock-app-tier
|
||||
title: Unblock the app tier — my Certificate stalled apps-wave-0, and the advertised https URL had no servable certificate
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
**I broke the app tier with PR #48 and this reverts the part that did it.** Two faults, both mine,
|
||||
both measured after the merge:
|
||||
|
||||
1. **Kyverno blocks cert-manager in app namespaces.** The temporary private-key Secrets
|
||||
(`plex-vip-<random>`) were denied by `secret-provenance/secrets-must-be-flux-managed`:
|
||||
|
||||
resource Secret/plex/plex-vip-rp4qj was blocked due to the following policies
|
||||
secret-provenance:
|
||||
secrets-must-be-flux-managed: Secrets in application namespaces must be delivered by
|
||||
Flux from SOPS-encrypted git ... or be exempt controller-generated Secrets.
|
||||
|
||||
The policy matches every Secret in a namespace labelled
|
||||
`homelab.keithjackson.dev/app-namespace: true` and excludes only Secrets labelled
|
||||
`homelab.keithjackson.dev/secret-managed-by: controller` or carrying a Flux label. **The `plex`
|
||||
namespace has that label; the `gateway` namespace — where the working wildcard cert lives — does
|
||||
NOT.** That asymmetry is why the existing Certificate was never affected and mine was, and I did
|
||||
not check it. cert-manager's temp key Secrets carry neither exempting label, and `secretTemplate`
|
||||
cannot label them (the keymanager controller creates them, not the Certificate template). Result:
|
||||
**no CertificateRequest, Order or Challenge was ever created.**
|
||||
|
||||
2. **Flux health-checks the Certificate — guarding the pod was not enough.** `optional: true` did
|
||||
exactly what it was designed to do: Plex rolled and ran fine without the Secret (verified). But
|
||||
`apps-wave-0` is `wait: true`, so Flux waits on every applied object with a known health type:
|
||||
|
||||
health check failed after 5m0.045564329s: timeout waiting for:
|
||||
[Certificate/plex/plex-vip status: 'InProgress']
|
||||
|
||||
apps-wave-0 never went Ready, and apps-wave-1/-2 plus everything downstream of platform-config
|
||||
sat in `DependencyNotReady` behind it. I reasoned about the pod and missed the object.
|
||||
|
||||
A third, independent problem the same PR introduced: **the advertised URL was `https://` with no
|
||||
servable certificate.** Plex answers 32400 with its `*.<serverhash>.plex.direct` cert, which cannot
|
||||
cover `plex-vip.${BASE_DOMAIN}` — the precise failure mode of the outage earlier that day.
|
||||
|
||||
- **`apps/wave-0/plex-cert.yaml` DELETED**, and `plex-cert.yaml` removed from
|
||||
`apps/wave-0/kustomization.yaml` `resources`, so Flux prunes the stuck Certificate and the health
|
||||
gate clears. **Deleted rather than merely unregistered** — my first attempt kept the file as
|
||||
"tracked but not onboarded" (the shape `keycloak.yaml` and `vm-demo.yaml` use) and **the repo's own
|
||||
gate rejected it**: `tls-lifecycle-automated` (R-TLS-02) fails any Certificate in the tree that no
|
||||
Flux path reconciles — *"nothing reconciles it, so a deleted Certificate/issuer/controller would
|
||||
stay deleted until a human re-applied it."* That pattern is legal for a Deployment and not for a
|
||||
TLS object, which is a distinction I had not known.
|
||||
- **`catalog/render.py`:** the now-dangling `plex-cert` EXTRA_UNITS entry removed.
|
||||
- **The re-enable recipe is recorded at the call site** — in `catalog/plex.yaml` beside the scheme
|
||||
it is coupled to — rather than in the deleted file: the Kyverno exemption, the non-gating
|
||||
Kustomization, and the `https` flip, as one change.
|
||||
- **`catalog/plex.yaml`:** `ADVERTISE_IP` → **`http://`**`plex-vip.${BASE_DOMAIN}:32400`, explicitly
|
||||
temporary, with the restore condition written at the call site.
|
||||
|
||||
**Everything else from PR #48 is kept and working** — the hostname design, the PiHole split-horizon
|
||||
answer, the `LanNetworksBandwidth` hook, the `optional` schema field, and the TLS env (inert without
|
||||
the Secret, so re-enabling is additive).
|
||||
|
||||
## What I verified
|
||||
|
||||
Live prod, 2026-10-04, read-only, after the #48/#50 merges.
|
||||
|
||||
- **The stall and its exact cause**: the health-check timeout message above, and the full dependency
|
||||
cascade (`cilium` → "dependency 'flux-system/flux-system' revision is not up to date", everything
|
||||
else `DependencyNotReady`). The root `cilium` wait was a normal revision-gated cascade; the
|
||||
*Certificate* was the real blocker.
|
||||
- **The Kyverno denial text, verbatim**, from the cert-manager controller log, repeating every ~30s
|
||||
across at least 15 distinct Secret names.
|
||||
- **The policy's own structure**, read from `platform/kyverno/admission/secret-provenance.yaml`:
|
||||
match = Secrets in `app-namespace: true` namespaces; exclude = `secret-managed-by: controller` OR
|
||||
a `kustomize.toolkit.fluxcd.io/name` label; validate = unconditional `deny`.
|
||||
- **The namespace-label asymmetry**: `plex` carries
|
||||
`homelab.keithjackson.dev/app-namespace: "true"`; `gateway` does not.
|
||||
- **cert-manager's controller ServiceAccount is `cert-manager`** in namespace `cert-manager` — the
|
||||
identity a future exemption should name.
|
||||
- **`optional: true` worked**: the new pod (`plex-c4f87c84d-zgjq5`) is `1/1 Running` with no Secret
|
||||
present, and `/config/certs` was never created.
|
||||
- **The postStart hook RAN and took effect** — the thing the whole hook was written for:
|
||||
`LanNetworksBandwidth="10.0.0.0/8,192.168.0.0/16"` is now set in the live `Preferences.xml`, and
|
||||
`customConnections="https://plex-vip.home.keithjackson.dev:32400"` was applied from the env. The
|
||||
certificate prefs are correctly **ABSENT**, i.e. the hook's "no bundle → leave them alone" guard
|
||||
behaved as designed.
|
||||
- **PiHole's split-horizon answer is correct**, asked of `pihole-0` directly:
|
||||
`plex-vip.home.keithjackson.dev -> 192.168.2.24`, alongside `plex -> .20` and `crafty-mc -> .22`,
|
||||
and the live `/etc/pihole/dnsmasq.conf` carries
|
||||
`address=/plex-vip.home.keithjackson.dev/192.168.2.24`. **This corrects my own earlier check**,
|
||||
which read `.20` and which I nearly reported as a failure — that test resolved via CLUSTER DNS
|
||||
(CoreDNS) from inside a pod and never consulted PiHole at all.
|
||||
- **The gate caught my first attempt and I fixed it rather than working around it.** Keeping the
|
||||
file unregistered produced `MERGE GATE: BLOCKED` (rc 1) with
|
||||
`CHECK FAIL (tls-lifecycle-automated): 1 of 612 assertions failed`. After deleting the file and
|
||||
the dangling EXTRA_UNITS entry: **rc 3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`,
|
||||
`render-golden`, `l1` and `approval-policy` all **pass** — the rc 3 being only the two known
|
||||
environmental skips (`gitea-settings`, `conformance` L2/L3). **Not green. Reported as rc 3.**
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **Whether this actually unblocks the tier.** Unreconciled at time of writing. The reasoning is
|
||||
that pruning the Certificate removes the only failing health check, but I have not seen
|
||||
apps-wave-0 go Ready.
|
||||
- **Whether `http://` restores client connectivity.** PiHole resolves the name and Plex serves 32400
|
||||
(both measured), but I have not watched a Plex client complete a connection over the advertised
|
||||
URL, and I did not re-check whether plex.tv publishes it.
|
||||
- **THE REMOTE BADGE — the symptom that started this.** `LanNetworksBandwidth` is set, but the log
|
||||
still tags traffic `Allowed Network (WAN)`: **82 WAN, 0 LAN**, with entries continuing after the
|
||||
pref was written. So either Plex needs a restart for it to take effect, or that log tag is not
|
||||
driven by this pref at all. **I do not know which, and the user-visible badge is the only real
|
||||
test** — it is in the Plex client, which I cannot see.
|
||||
- **The Hubble flow export is NOT active.** The cilium agents are still the pre-merge pods, so the
|
||||
`filePath: stdout` question is still open and no flow has been exported. The Cilium rollout — and
|
||||
its LAN-DNS risk, since prod-3 holds the Gateway, PiHole-DNS and Plex VIPs — is still AHEAD.
|
||||
- **Whether a cert-manager exemption is the right shape**, or whether it widens
|
||||
`secret-provenance` more than the operator wants. It is `platform/kyverno/**`, a sensitive path,
|
||||
and deliberately not attempted here.
|
||||
- **Whether the pruned Certificate leaves residue** — I did not check for orphaned
|
||||
`plex-vip-<random>` Secrets or CertificateRequests after the prune. Note Kyverno DENIED every one
|
||||
of those Secrets, so there may be nothing to clean, but that is an inference, not a check.
|
||||
- **Whether `tls-lifecycle-automated` would also have objected to the Certificate living in a
|
||||
`wait: false` Kustomization** (the recommended re-enable shape). It demands a reconciling Flux
|
||||
path, which that provides, but I did not test it.
|
||||
- **The unencrypted advertised URL.** `http://` is tolerable only while remote access does not work
|
||||
(no WAN forward). Nothing enforces that assumption, and nothing will complain if `http` is still
|
||||
there the day the port-forward appears.
|
||||
Reference in new issue
Block a user