The 7 falsely-firing platform alerts (KeycloakDown, KeycloakDatabaseDown,
KubeAPIDown, KubeletDown, VPNTunnelDown, KubeVirtControlPlaneDown,
CDINoDefaultStorageClass) share ONE root cause, not per-rule label mismatches:
the Prometheus PVC is 100% full ("no space left on device" on every WAL commit).
Evidence from the live dev cluster:
- /prometheus is 4.8G/5Gi, 0 bytes free; every scrape logs "Scrape commit
failed ... write to WAL: no space left on device". Scrapes still return 200
(targets show healthy, lastScrape=now) but samples are never stored, so the
TSDB head froze at 2026-08-28 13:45 UTC while the pod clock reads ~19:33 UTC.
- Instant queries at "now" therefore return an EMPTY vector, so every
absent()/==0 platform rule trips. Evaluated at the last-good-sample time each
expr is correctly FALSE (keycloak-0/keycloak-db-1 ready=1, apiserver+kubelet
up=1 x3, tun0 series present, kubevirt control-plane expr empty, CDI default
present) — the rules are correct; the monitor was simply blind.
`retention` alone prunes only by AGE: this cluster is 3d2h old with 3d dev
retention, so time-pruning has NEVER run and 3d of block churn (~1.2Gi/day) plus
a WAL that ballooned past 1Gi overran the 5Gi disk. Fix adds a size-based
retention cap (new MONITORING_PROMETHEUS_RETENTION_SIZE knob) kept below each
env's PVC so Prometheus drops oldest blocks before the disk fills, and bumps the
dev PVC 5Gi->8Gi for real WAL/head/compaction headroom.
- platform/monitoring/kube-prometheus-stack.yaml: retentionSize: ${knob}
- clusters/base/cluster-vars.yaml: knob default 8GiB (base 10Gi PVC)
- clusters/dev/kustomization.yaml: PVC 5Gi->8Gi, retentionSize 5GiB
- clusters/prod/kustomization.yaml: retentionSize 45GiB (50Gi PVC)
- platform/monitoring/README.md: knob table + rationale
Deploy note (PM): the PVC is already 100% full, so Prometheus cannot self-prune
until it can write. The 5Gi->8Gi bump must land as a Longhorn online expansion
(operator PVC expansion / manual patch); once free space returns, retentionSize
keeps it healthy. Watchdog left untouched.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Grafana's pod predated the grafana-oidc-credentials secret, so generic_oauth started with
an empty client_secret (disabled) → /login/generic_oauth didn't redirect to Keycloak.
Add a podAnnotation to force one roll; bump on future OIDC secret changes.
prometheus-...-0 was OOMKilled (exit 137) every ~4min during TSDB compaction/checkpoint
(261 restarts/37h). Raise limit to 2Gi + request to 512Mi; nodes have headroom.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Enable monitoring in platform/kustomization.yaml (Phase 4). Grafana admin +
Alertmanager SMTP secrets are provisioned (SOPS) and wired; Longhorn SC is up for
Prometheus/Loki PVCs. Set the Alertmanager receiver 'to:' to a valid inbox
([email protected]) so the config loads (REPLACE-ME is not a valid address);
operator can change it. NOTE: smtp_from ([email protected]) differs from
the Gmail auth user — Gmail may reject/rewrite unless a send-as alias is configured;
that only affects email delivery at alert time, not stack health.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Prep only — nothing deployed; running cluster untouched.
Reconciliation graph (operator-approved Option A): author clusters/dev/rollout/
= 4 per-component Flux Kustomizations (kyverno->cilium->longhorn->monitoring,
monitoring dependsOn longhorn). Dev overlay $patch:delete's the monolithic base
platform/apps and adds ./rollout, so only the no-secret tier reconciles; the
secret-blocked components are simply not created until hand-offs arrive. The
no-secret tier carries no SOPS content, so it needs no sops-age key. prod is
untouched (still builds the monolith). Convergence path coded in (delete rollout/
+ the two deletes) — R-ENV-02 divergence flagged for architect review.
Component hardening via cluster-vars (prod keeps strict posture):
- KYVERNO_VALIDATION_ACTION Enforce->Audit(dev), _WEBHOOK_FAILURE_POLICY
Fail->Ignore(dev), _ADMISSION_REPLICAS 3->1(dev); webhook namespaceSelector
excludes system+platform namespaces.
- MONITORING_STORAGE_CLASS var; GRAFANA_ADMIN_SECRET var (dev ''=chart default);
Alertmanager SMTP/secret deferred to Phase-4 with verbatim re-enable markers.
Cilium parity verified against the live CNI (no-op adopt). Validated:
kubectl kustomize dev/prod/all-components, envsubst->YAML parse, make -C ci gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
jakman85andClaude Opus 4.8