fix(kubevirt): drop broken http CDI ServiceMonitor causing false CDIOperatorDown
The cdi-operator manages its own `service-monitor-cdi` (scheme https + cdi-metrics-reader-token) which backs CDI's shipped recording rules (kubevirt_cdi_operator_up / cluster:kubevirt_cdi_operator_up:sum) and the CDIOperatorDown alert. Our extra `cdi` ServiceMonitor scraped the same cdi-prometheus-metrics service (HTTPS :8443) over http, so every scrape failed with 400 Bad Request, adding down (up=0) targets for the CDI pods. That poisoned the up-aggregation the recording rule sums, driving it to 0 and firing a false CDIOperatorDown even though the operator was Running, Ready, and Available=True. Remove the duplicate; keep the kubevirt SM (virt-operator ships none of its own). Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
1 parent
948c390272
commit
6a7e9d15e7
1 file changed
+8
-22
@@ -8,8 +8,14 @@
|
|||||||
# Prometheus discovers ServiceMonitors cluster-wide (serviceMonitorSelectorNilUsesHelmValues:false in
|
# Prometheus discovers ServiceMonitors cluster-wide (serviceMonitorSelectorNilUsesHelmValues:false in
|
||||||
# platform/monitoring/kube-prometheus-stack.yaml), so these can live beside their target Services.
|
# platform/monitoring/kube-prometheus-stack.yaml), so these can live beside their target Services.
|
||||||
# NOTE (verify at bring-up, same convention as alerts.yaml): KubeVirt serves metrics over HTTPS with a
|
# NOTE (verify at bring-up, same convention as alerts.yaml): KubeVirt serves metrics over HTTPS with a
|
||||||
# self-signed internal cert (insecureSkipVerify) on the port named `metrics`; CDI serves plain HTTP on
|
# self-signed internal cert (insecureSkipVerify) on the port named `metrics`. virt-operator does NOT
|
||||||
# its `metrics` port. Port names/scheme are confirmed against the deployed Services on first reconcile.
|
# ship its own ServiceMonitor, so this one is required.
|
||||||
|
#
|
||||||
|
# CDI is intentionally NOT scraped here: the cdi-operator manages its own `service-monitor-cdi`
|
||||||
|
# (scheme https + a cdi-metrics-reader-token, owned by the cdi-operator Deployment) which backs CDI's
|
||||||
|
# shipped recording rules (kubevirt_cdi_operator_up / cluster:kubevirt_cdi_operator_up:sum) and the
|
||||||
|
# CDIOperatorDown alert. A second http ServiceMonitor here added down (400) targets for the same pods,
|
||||||
|
# poisoning that up-aggregation to 0 and firing a false CDIOperatorDown — so it was removed.
|
||||||
apiVersion: monitoring.coreos.com/v1
|
apiVersion: monitoring.coreos.com/v1
|
||||||
kind: ServiceMonitor
|
kind: ServiceMonitor
|
||||||
metadata:
|
metadata:
|
||||||
@@ -31,23 +37,3 @@ spec:
|
|||||||
insecureSkipVerify: true
|
insecureSkipVerify: true
|
||||||
honorLabels: true
|
honorLabels: true
|
||||||
interval: 30s
|
interval: 30s
|
||||||
---
|
|
||||||
apiVersion: monitoring.coreos.com/v1
|
|
||||||
kind: ServiceMonitor
|
|
||||||
metadata:
|
|
||||||
name: cdi
|
|
||||||
namespace: cdi
|
|
||||||
labels:
|
|
||||||
release: kube-prometheus-stack
|
|
||||||
spec:
|
|
||||||
selector:
|
|
||||||
matchLabels:
|
|
||||||
prometheus.cdi.kubevirt.io: "true"
|
|
||||||
namespaceSelector:
|
|
||||||
matchNames:
|
|
||||||
- cdi
|
|
||||||
endpoints:
|
|
||||||
- port: metrics
|
|
||||||
scheme: http
|
|
||||||
honorLabels: true
|
|
||||||
interval: 30s
|
|
||||||
Reference in new issue
Block a user