From 8b3624264347a1534146b3c3e21e0053e98ff5c1 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 13 Sep 2026 16:32:28 -0500 Subject: [PATCH] =?UTF-8?q?fix(monitoring):=20kube-state-metrics=20was=20O?= =?UTF-8?q?OMKilled=2091=20times=20=E2=80=94=2013=20rules=20went=20intermi?= =?UTF-8?q?ttent?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Working set measured at 111 MiB with a 6h peak of 123.3 MiB against a 128Mi limit, so it died on every spike — exit 137, roughly every 11 minutes, 91 restarts in 16h. The consequence is worse than the restarts. Thirteen alert rules read kube_* series, so each kill made them intermittently unable to fire, which is the quiet version of the inert-alert class found elsewhere today; KubePodNotReady and KubeDeploymentReplicasMismatch were both sitting pending on it. Self-inflicted: enabling customResourceState for the Velero Schedule metrics earlier today added a CRD informer plus the mandatory CRD list/watch, and that marginal cost crossed a limit already sitting about 4 MiB above the working set. Raised to 384Mi, roughly 3x the observed peak, which costs nothing on nodes reporting 18.7-22.3 GiB available. Co-Authored-By: Claude Opus 5 --- platform/monitoring/kube-prometheus-stack.yaml | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/platform/monitoring/kube-prometheus-stack.yaml b/platform/monitoring/kube-prometheus-stack.yaml index da34233..53019cb 100644 --- a/platform/monitoring/kube-prometheus-stack.yaml +++ b/platform/monitoring/kube-prometheus-stack.yaml @@ -328,12 +328,25 @@ spec: labelsFromPath: cron: [spec, schedule] phase: [status, phase] + # MEMORY RAISED 2026-09-13 after 91 OOMKills. Measured, not guessed: working set 111 MiB + # with a 6h peak of 123.3 MiB against the old 128Mi limit — KSM was dying on every spike, + # roughly every 11 minutes (exit 137). That matters beyond the restarts: THIRTEEN alert + # rules read kube_* series, so each kill made them intermittently inert, and an alert that + # is only sometimes able to fire is the quiet version of the inert-alert class this repo + # keeps getting bitten by (KubePodNotReady and KubeDeploymentReplicasMismatch were both + # stuck pending on it). The trigger was enabling customResourceState below for the Velero + # Schedule metrics — it adds a CRD informer plus the mandatory CRD list/watch, and that + # marginal cost crossed a limit already sitting ~4 MiB above the working set. + # 384Mi is ~3x the observed peak and costs nothing here: the nodes report 18.7-22.3 GiB + # available. Re-measure with + # max_over_time(container_memory_working_set_bytes{container="kube-state-metrics"}[6h]) + # before trimming it back. resources: requests: cpu: 20m - memory: 64Mi + memory: 192Mi limits: - memory: 128Mi + memory: 384Mi prometheus-node-exporter: resources: requests: