fix(monitoring): cap Prometheus TSDB by size to stop false absent()/==0 alerts

The 7 falsely-firing platform alerts (KeycloakDown, KeycloakDatabaseDown,
KubeAPIDown, KubeletDown, VPNTunnelDown, KubeVirtControlPlaneDown,
CDINoDefaultStorageClass) share ONE root cause, not per-rule label mismatches:
the Prometheus PVC is 100% full ("no space left on device" on every WAL commit).

Evidence from the live dev cluster:
- /prometheus is 4.8G/5Gi, 0 bytes free; every scrape logs "Scrape commit
  failed ... write to WAL: no space left on device". Scrapes still return 200
  (targets show healthy, lastScrape=now) but samples are never stored, so the
  TSDB head froze at 2026-08-28 13:45 UTC while the pod clock reads ~19:33 UTC.
- Instant queries at "now" therefore return an EMPTY vector, so every
  absent()/==0 platform rule trips. Evaluated at the last-good-sample time each
  expr is correctly FALSE (keycloak-0/keycloak-db-1 ready=1, apiserver+kubelet
  up=1 x3, tun0 series present, kubevirt control-plane expr empty, CDI default
  present) — the rules are correct; the monitor was simply blind.

`retention` alone prunes only by AGE: this cluster is 3d2h old with 3d dev
retention, so time-pruning has NEVER run and 3d of block churn (~1.2Gi/day) plus
a WAL that ballooned past 1Gi overran the 5Gi disk. Fix adds a size-based
retention cap (new MONITORING_PROMETHEUS_RETENTION_SIZE knob) kept below each
env's PVC so Prometheus drops oldest blocks before the disk fills, and bumps the
dev PVC 5Gi->8Gi for real WAL/head/compaction headroom.

- platform/monitoring/kube-prometheus-stack.yaml: retentionSize: ${knob}
- clusters/base/cluster-vars.yaml: knob default 8GiB (base 10Gi PVC)
- clusters/dev/kustomization.yaml: PVC 5Gi->8Gi, retentionSize 5GiB
- clusters/prod/kustomization.yaml: retentionSize 45GiB (50Gi PVC)
- platform/monitoring/README.md: knob table + rationale

Deploy note (PM): the PVC is already 100% full, so Prometheus cannot self-prune
until it can write. The 5Gi->8Gi bump must land as a Longhorn online expansion
(operator PVC expansion / manual patch); once free space returns, retentionSize
keeps it healthy. Watchdog left untouched.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

platform/monitoring/kube-prometheus-stack.yaml

@@ -45,6 +45,17 @@ spec:
       prometheusSpec:
         replicas: 1
         retention: ${MONITORING_PROMETHEUS_RETENTION}     # overlay knob (dev 3d / prod long)
+        # SIZE-based retention cap (docs/10, R-OBS). `retention` alone prunes only by AGE — on a young
+        # cluster (< retention window) NOTHING is pruned yet, so a busy TSDB fills the PVC before the
+        # time window elapses. A full /prometheus wedges ALL WAL commits ("no space left on device"):
+        # scrapes still 200 but samples are never stored, the head freezes, and every absent()/==0
+        # platform alert (Keycloak/KubeAPI/Kubelet/VPN/KubeVirt/CDI…) fires falsely because instant
+        # queries at "now" return empty. retentionSize makes Prometheus drop the OLDEST blocks once the
+        # block set approaches this cap, so the disk never fills regardless of cluster age or churn.
+        # Kept BELOW the PVC size (leaves headroom for the WAL + head + compaction scratch, which are
+        # NOT counted by retentionSize). Prometheus size units are *iB (GiB), distinct from the K8s PVC
+        # `Gi`, so it is a separate overlay knob rather than a reuse of the PVC-size knob.
+        retentionSize: ${MONITORING_PROMETHEUS_RETENTION_SIZE}   # overlay knob (dev 5GiB / base 8GiB / prod 45GiB)
         # Discover ServiceMonitors/PodMonitors/Rules from ALL namespaces (flux, cert-manager,
         # longhorn, velero, keycloak, cilium/hubble ship their own) — not just this release's.
         ruleSelectorNilUsesHelmValues: false