fix(monitoring): cap Prometheus TSDB by size to stop false absent()/==0 alerts

The 7 falsely-firing platform alerts (KeycloakDown, KeycloakDatabaseDown,
KubeAPIDown, KubeletDown, VPNTunnelDown, KubeVirtControlPlaneDown,
CDINoDefaultStorageClass) share ONE root cause, not per-rule label mismatches:
the Prometheus PVC is 100% full ("no space left on device" on every WAL commit).

Evidence from the live dev cluster:
- /prometheus is 4.8G/5Gi, 0 bytes free; every scrape logs "Scrape commit
  failed ... write to WAL: no space left on device". Scrapes still return 200
  (targets show healthy, lastScrape=now) but samples are never stored, so the
  TSDB head froze at 2026-08-28 13:45 UTC while the pod clock reads ~19:33 UTC.
- Instant queries at "now" therefore return an EMPTY vector, so every
  absent()/==0 platform rule trips. Evaluated at the last-good-sample time each
  expr is correctly FALSE (keycloak-0/keycloak-db-1 ready=1, apiserver+kubelet
  up=1 x3, tun0 series present, kubevirt control-plane expr empty, CDI default
  present) — the rules are correct; the monitor was simply blind.

`retention` alone prunes only by AGE: this cluster is 3d2h old with 3d dev
retention, so time-pruning has NEVER run and 3d of block churn (~1.2Gi/day) plus
a WAL that ballooned past 1Gi overran the 5Gi disk. Fix adds a size-based
retention cap (new MONITORING_PROMETHEUS_RETENTION_SIZE knob) kept below each
env's PVC so Prometheus drops oldest blocks before the disk fills, and bumps the
dev PVC 5Gi->8Gi for real WAL/head/compaction headroom.

- platform/monitoring/kube-prometheus-stack.yaml: retentionSize: ${knob}
- clusters/base/cluster-vars.yaml: knob default 8GiB (base 10Gi PVC)
- clusters/dev/kustomization.yaml: PVC 5Gi->8Gi, retentionSize 5GiB
- clusters/prod/kustomization.yaml: retentionSize 45GiB (50Gi PVC)
- platform/monitoring/README.md: knob table + rationale

Deploy note (PM): the PVC is already 100% full, so Prometheus cannot self-prune
until it can write. The 5Gi->8Gi bump must land as a Longhorn online expansion
(operator PVC expansion / manual patch); once free space returns, retentionSize
keeps it healthy. Watchdog left untouched.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

clusters/base/cluster-vars.yaml

@@ -35,6 +35,11 @@ data:
   # --- observability sizing (docs/10) ---
   MONITORING_PROMETHEUS_RETENTION: "7d"     # dev short | prod long
   MONITORING_PROMETHEUS_PVC_SIZE: "10Gi"
+  # SIZE-based retention cap for Prometheus (Prometheus *iB units, NOT the K8s `Gi` of the PVC knob).
+  # MUST stay below MONITORING_PROMETHEUS_PVC_SIZE — leaves headroom for the WAL + head + compaction
+  # scratch (retentionSize counts only persisted blocks). Prevents a busy TSDB from filling /prometheus
+  # (a full disk wedges all WAL commits and makes every absent()/==0 alert fire falsely). Base 10Gi PVC.
+  MONITORING_PROMETHEUS_RETENTION_SIZE: "8GiB"
   MONITORING_LOKI_PVC_SIZE: "10Gi"
 
   # --- admission enforcement (Kyverno, ADR-0011). Base defaults are the STRICT/prod posture;

clusters/dev/kustomization.yaml

@@ -134,9 +134,17 @@ patches:
       - op: replace
         path: /data/MONITORING_PROMETHEUS_RETENTION
         value: "3d"
+      # 5Gi was too tight: 3d of this cluster's series volume (~1.2Gi/day of blocks) plus a WAL that
+      # ballooned past 1Gi overran the disk, wedging WAL commits and firing every absent()/==0 alert
+      # falsely. Bumped to 8Gi (still lighter than base 10Gi) so the size-cap below has real headroom.
       - op: replace
         path: /data/MONITORING_PROMETHEUS_PVC_SIZE
-        value: "5Gi"
+        value: "8Gi"
+      # Size-cap kept well below the 8Gi PVC (headroom for WAL + head + compaction scratch) so
+      # Prometheus prunes oldest blocks by size and never fills the disk on this young cluster.
+      - op: replace
+        path: /data/MONITORING_PROMETHEUS_RETENTION_SIZE
+        value: "5GiB"
       - op: replace
         path: /data/MONITORING_LOKI_PVC_SIZE
         value: "5Gi"

clusters/prod/kustomization.yaml

@@ -31,6 +31,11 @@ patches:
       - op: replace
         path: /data/MONITORING_PROMETHEUS_PVC_SIZE
         value: "50Gi"
+      # Size-cap below the 50Gi PVC (headroom for WAL + head + compaction scratch): prod may hit the
+      # size cap before 30d if series volume is high, keeping the disk from filling either way.
+      - op: replace
+        path: /data/MONITORING_PROMETHEUS_RETENTION_SIZE
+        value: "45GiB"
       - op: replace
         path: /data/MONITORING_LOKI_PVC_SIZE
         value: "50Gi"

platform/monitoring/README.md

@@ -90,9 +90,17 @@ Only three substitution vars touch this subtree; the dev overlay (`clusters/dev`
 | Var | base | dev |
 |---|---|---|
 | `${MONITORING_PROMETHEUS_RETENTION}` | `7d` | `3d` |
-| `${MONITORING_PROMETHEUS_PVC_SIZE}` | `10Gi` | `5Gi` |
+| `${MONITORING_PROMETHEUS_PVC_SIZE}` | `10Gi` | `8Gi` |
+| `${MONITORING_PROMETHEUS_RETENTION_SIZE}` | `8GiB` | `5GiB` |
 | `${MONITORING_LOKI_PVC_SIZE}` | `10Gi` | `5Gi` |
 
+`MONITORING_PROMETHEUS_RETENTION_SIZE` is a **size-based** retention cap (Prometheus `*iB` units,
+distinct from the K8s `Gi` PVC knob) and must stay **below** `MONITORING_PROMETHEUS_PVC_SIZE`. Time-only
+retention prunes nothing until the age window elapses, so a young/busy cluster fills `/prometheus` first;
+a full disk wedges every WAL commit (scrapes still 200 but samples are dropped, the head freezes) and
+makes every `absent()`/`==0` platform alert fire falsely. The size cap drops the oldest blocks before the
+disk fills. (prod overlay: PVC `50Gi`, cap `45GiB`.)
+
 PVCs use the replicated `longhorn` StorageClass (ADR-0003).
 
 ## Notes / known gaps for bring-up

platform/monitoring/kube-prometheus-stack.yaml

@@ -45,6 +45,17 @@ spec:
       prometheusSpec:
         replicas: 1
         retention: ${MONITORING_PROMETHEUS_RETENTION}     # overlay knob (dev 3d / prod long)
+        # SIZE-based retention cap (docs/10, R-OBS). `retention` alone prunes only by AGE — on a young
+        # cluster (< retention window) NOTHING is pruned yet, so a busy TSDB fills the PVC before the
+        # time window elapses. A full /prometheus wedges ALL WAL commits ("no space left on device"):
+        # scrapes still 200 but samples are never stored, the head freezes, and every absent()/==0
+        # platform alert (Keycloak/KubeAPI/Kubelet/VPN/KubeVirt/CDI…) fires falsely because instant
+        # queries at "now" return empty. retentionSize makes Prometheus drop the OLDEST blocks once the
+        # block set approaches this cap, so the disk never fills regardless of cluster age or churn.
+        # Kept BELOW the PVC size (leaves headroom for the WAL + head + compaction scratch, which are
+        # NOT counted by retentionSize). Prometheus size units are *iB (GiB), distinct from the K8s PVC
+        # `Gi`, so it is a separate overlay knob rather than a reuse of the PVC-size knob.
+        retentionSize: ${MONITORING_PROMETHEUS_RETENTION_SIZE}   # overlay knob (dev 5GiB / base 8GiB / prod 45GiB)
         # Discover ServiceMonitors/PodMonitors/Rules from ALL namespaces (flux, cert-manager,
         # longhorn, velero, keycloak, cilium/hubble ship their own) — not just this release's.
         ruleSelectorNilUsesHelmValues: false