From eb31d2ef58a595b195a9b45671e386ca0492ccb5 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Fri, 28 Aug 2026 14:38:12 -0500 Subject: [PATCH] fix(monitoring): cap Prometheus TSDB by size to stop false absent()/==0 alerts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 7 falsely-firing platform alerts (KeycloakDown, KeycloakDatabaseDown, KubeAPIDown, KubeletDown, VPNTunnelDown, KubeVirtControlPlaneDown, CDINoDefaultStorageClass) share ONE root cause, not per-rule label mismatches: the Prometheus PVC is 100% full ("no space left on device" on every WAL commit). Evidence from the live dev cluster: - /prometheus is 4.8G/5Gi, 0 bytes free; every scrape logs "Scrape commit failed ... write to WAL: no space left on device". Scrapes still return 200 (targets show healthy, lastScrape=now) but samples are never stored, so the TSDB head froze at 2026-08-28 13:45 UTC while the pod clock reads ~19:33 UTC. - Instant queries at "now" therefore return an EMPTY vector, so every absent()/==0 platform rule trips. Evaluated at the last-good-sample time each expr is correctly FALSE (keycloak-0/keycloak-db-1 ready=1, apiserver+kubelet up=1 x3, tun0 series present, kubevirt control-plane expr empty, CDI default present) — the rules are correct; the monitor was simply blind. `retention` alone prunes only by AGE: this cluster is 3d2h old with 3d dev retention, so time-pruning has NEVER run and 3d of block churn (~1.2Gi/day) plus a WAL that ballooned past 1Gi overran the 5Gi disk. Fix adds a size-based retention cap (new MONITORING_PROMETHEUS_RETENTION_SIZE knob) kept below each env's PVC so Prometheus drops oldest blocks before the disk fills, and bumps the dev PVC 5Gi->8Gi for real WAL/head/compaction headroom. - platform/monitoring/kube-prometheus-stack.yaml: retentionSize: ${knob} - clusters/base/cluster-vars.yaml: knob default 8GiB (base 10Gi PVC) - clusters/dev/kustomization.yaml: PVC 5Gi->8Gi, retentionSize 5GiB - clusters/prod/kustomization.yaml: retentionSize 45GiB (50Gi PVC) - platform/monitoring/README.md: knob table + rationale Deploy note (PM): the PVC is already 100% full, so Prometheus cannot self-prune until it can write. The 5Gi->8Gi bump must land as a Longhorn online expansion (operator PVC expansion / manual patch); once free space returns, retentionSize keeps it healthy. Watchdog left untouched. Co-Authored-By: Claude Opus 4.8 --- clusters/base/cluster-vars.yaml | 5 +++++ clusters/dev/kustomization.yaml | 10 +++++++++- clusters/prod/kustomization.yaml | 5 +++++ platform/monitoring/README.md | 10 +++++++++- platform/monitoring/kube-prometheus-stack.yaml | 11 +++++++++++ 5 files changed, 39 insertions(+), 2 deletions(-) diff --git a/clusters/base/cluster-vars.yaml b/clusters/base/cluster-vars.yaml index b2f0bf9..5a1ba0f 100644 --- a/clusters/base/cluster-vars.yaml +++ b/clusters/base/cluster-vars.yaml @@ -35,6 +35,11 @@ data: # --- observability sizing (docs/10) --- MONITORING_PROMETHEUS_RETENTION: "7d" # dev short | prod long MONITORING_PROMETHEUS_PVC_SIZE: "10Gi" + # SIZE-based retention cap for Prometheus (Prometheus *iB units, NOT the K8s `Gi` of the PVC knob). + # MUST stay below MONITORING_PROMETHEUS_PVC_SIZE — leaves headroom for the WAL + head + compaction + # scratch (retentionSize counts only persisted blocks). Prevents a busy TSDB from filling /prometheus + # (a full disk wedges all WAL commits and makes every absent()/==0 alert fire falsely). Base 10Gi PVC. + MONITORING_PROMETHEUS_RETENTION_SIZE: "8GiB" MONITORING_LOKI_PVC_SIZE: "10Gi" # --- admission enforcement (Kyverno, ADR-0011). Base defaults are the STRICT/prod posture; diff --git a/clusters/dev/kustomization.yaml b/clusters/dev/kustomization.yaml index 511c662..ec227fc 100644 --- a/clusters/dev/kustomization.yaml +++ b/clusters/dev/kustomization.yaml @@ -134,9 +134,17 @@ patches: - op: replace path: /data/MONITORING_PROMETHEUS_RETENTION value: "3d" + # 5Gi was too tight: 3d of this cluster's series volume (~1.2Gi/day of blocks) plus a WAL that + # ballooned past 1Gi overran the disk, wedging WAL commits and firing every absent()/==0 alert + # falsely. Bumped to 8Gi (still lighter than base 10Gi) so the size-cap below has real headroom. - op: replace path: /data/MONITORING_PROMETHEUS_PVC_SIZE - value: "5Gi" + value: "8Gi" + # Size-cap kept well below the 8Gi PVC (headroom for WAL + head + compaction scratch) so + # Prometheus prunes oldest blocks by size and never fills the disk on this young cluster. + - op: replace + path: /data/MONITORING_PROMETHEUS_RETENTION_SIZE + value: "5GiB" - op: replace path: /data/MONITORING_LOKI_PVC_SIZE value: "5Gi" diff --git a/clusters/prod/kustomization.yaml b/clusters/prod/kustomization.yaml index c0902bd..3eaec4f 100644 --- a/clusters/prod/kustomization.yaml +++ b/clusters/prod/kustomization.yaml @@ -31,6 +31,11 @@ patches: - op: replace path: /data/MONITORING_PROMETHEUS_PVC_SIZE value: "50Gi" + # Size-cap below the 50Gi PVC (headroom for WAL + head + compaction scratch): prod may hit the + # size cap before 30d if series volume is high, keeping the disk from filling either way. + - op: replace + path: /data/MONITORING_PROMETHEUS_RETENTION_SIZE + value: "45GiB" - op: replace path: /data/MONITORING_LOKI_PVC_SIZE value: "50Gi" diff --git a/platform/monitoring/README.md b/platform/monitoring/README.md index bb494d7..2e24805 100644 --- a/platform/monitoring/README.md +++ b/platform/monitoring/README.md @@ -90,9 +90,17 @@ Only three substitution vars touch this subtree; the dev overlay (`clusters/dev` | Var | base | dev | |---|---|---| | `${MONITORING_PROMETHEUS_RETENTION}` | `7d` | `3d` | -| `${MONITORING_PROMETHEUS_PVC_SIZE}` | `10Gi` | `5Gi` | +| `${MONITORING_PROMETHEUS_PVC_SIZE}` | `10Gi` | `8Gi` | +| `${MONITORING_PROMETHEUS_RETENTION_SIZE}` | `8GiB` | `5GiB` | | `${MONITORING_LOKI_PVC_SIZE}` | `10Gi` | `5Gi` | +`MONITORING_PROMETHEUS_RETENTION_SIZE` is a **size-based** retention cap (Prometheus `*iB` units, +distinct from the K8s `Gi` PVC knob) and must stay **below** `MONITORING_PROMETHEUS_PVC_SIZE`. Time-only +retention prunes nothing until the age window elapses, so a young/busy cluster fills `/prometheus` first; +a full disk wedges every WAL commit (scrapes still 200 but samples are dropped, the head freezes) and +makes every `absent()`/`==0` platform alert fire falsely. The size cap drops the oldest blocks before the +disk fills. (prod overlay: PVC `50Gi`, cap `45GiB`.) + PVCs use the replicated `longhorn` StorageClass (ADR-0003). ## Notes / known gaps for bring-up diff --git a/platform/monitoring/kube-prometheus-stack.yaml b/platform/monitoring/kube-prometheus-stack.yaml index ec660b5..835e528 100644 --- a/platform/monitoring/kube-prometheus-stack.yaml +++ b/platform/monitoring/kube-prometheus-stack.yaml @@ -45,6 +45,17 @@ spec: prometheusSpec: replicas: 1 retention: ${MONITORING_PROMETHEUS_RETENTION} # overlay knob (dev 3d / prod long) + # SIZE-based retention cap (docs/10, R-OBS). `retention` alone prunes only by AGE — on a young + # cluster (< retention window) NOTHING is pruned yet, so a busy TSDB fills the PVC before the + # time window elapses. A full /prometheus wedges ALL WAL commits ("no space left on device"): + # scrapes still 200 but samples are never stored, the head freezes, and every absent()/==0 + # platform alert (Keycloak/KubeAPI/Kubelet/VPN/KubeVirt/CDI…) fires falsely because instant + # queries at "now" return empty. retentionSize makes Prometheus drop the OLDEST blocks once the + # block set approaches this cap, so the disk never fills regardless of cluster age or churn. + # Kept BELOW the PVC size (leaves headroom for the WAL + head + compaction scratch, which are + # NOT counted by retentionSize). Prometheus size units are *iB (GiB), distinct from the K8s PVC + # `Gi`, so it is a separate overlay knob rather than a reuse of the PVC-size knob. + retentionSize: ${MONITORING_PROMETHEUS_RETENTION_SIZE} # overlay knob (dev 5GiB / base 8GiB / prod 45GiB) # Discover ServiceMonitors/PodMonitors/Rules from ALL namespaces (flux, cert-manager, # longhorn, velero, keycloak, cilium/hubble ship their own) — not just this release's. ruleSelectorNilUsesHelmValues: false