jakman85andClaude Opus 4.8
eb31d2ef58
The 7 falsely-firing platform alerts (KeycloakDown, KeycloakDatabaseDown,
KubeAPIDown, KubeletDown, VPNTunnelDown, KubeVirtControlPlaneDown,
CDINoDefaultStorageClass) share ONE root cause, not per-rule label mismatches:
the Prometheus PVC is 100% full ("no space left on device" on every WAL commit).
Evidence from the live dev cluster:
- /prometheus is 4.8G/5Gi, 0 bytes free; every scrape logs "Scrape commit
failed ... write to WAL: no space left on device". Scrapes still return 200
(targets show healthy, lastScrape=now) but samples are never stored, so the
TSDB head froze at 2026-08-28 13:45 UTC while the pod clock reads ~19:33 UTC.
- Instant queries at "now" therefore return an EMPTY vector, so every
absent()/==0 platform rule trips. Evaluated at the last-good-sample time each
expr is correctly FALSE (keycloak-0/keycloak-db-1 ready=1, apiserver+kubelet
up=1 x3, tun0 series present, kubevirt control-plane expr empty, CDI default
present) — the rules are correct; the monitor was simply blind.
`retention` alone prunes only by AGE: this cluster is 3d2h old with 3d dev
retention, so time-pruning has NEVER run and 3d of block churn (~1.2Gi/day) plus
a WAL that ballooned past 1Gi overran the 5Gi disk. Fix adds a size-based
retention cap (new MONITORING_PROMETHEUS_RETENTION_SIZE knob) kept below each
env's PVC so Prometheus drops oldest blocks before the disk fills, and bumps the
dev PVC 5Gi->8Gi for real WAL/head/compaction headroom.
- platform/monitoring/kube-prometheus-stack.yaml: retentionSize: ${knob}
- clusters/base/cluster-vars.yaml: knob default 8GiB (base 10Gi PVC)
- clusters/dev/kustomization.yaml: PVC 5Gi->8Gi, retentionSize 5GiB
- clusters/prod/kustomization.yaml: retentionSize 45GiB (50Gi PVC)
- platform/monitoring/README.md: knob table + rationale
Deploy note (PM): the PVC is already 100% full, so Prometheus cannot self-prune
until it can write. The 5Gi->8Gi bump must land as a Longhorn online expansion
(operator PVC expansion / manual patch); once free space returns, retentionSize
keeps it healthy. Watchdog left untouched.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
76 lines
5.0 KiB
YAML
76 lines
5.0 KiB
YAML
---
|
|
# cluster-vars — the overlay knob surface (R-ENV-02).
|
|
#
|
|
# This ConfigMap holds BASE DEFAULTS. Flux `postBuild.substituteFrom` injects these into every
|
|
# `${TOKEN}` in platform/ and apps/ manifests at reconcile time. dev/ and prod/ overlays STRATEGIC-
|
|
# MERGE PATCH this single ConfigMap — that patch is the *entire* difference between environments.
|
|
#
|
|
# Add a knob here (with a base default) before referencing `${KNOB}` in a component manifest, then
|
|
# give each overlay its env-scoped value. Never fork a base manifest to differ per environment.
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: cluster-vars
|
|
namespace: flux-system
|
|
data:
|
|
# --- identity of the environment ---
|
|
CLUSTER_ENV: "base" # dev | prod (overlays set this)
|
|
|
|
# --- DNS / TLS ---
|
|
BASE_DOMAIN: "home.keithjackson.dev" # platform hostname suffix (same both envs; split-horizon)
|
|
DNS_ZONE: "keithjackson.dev" # the actual Cloudflare ZONE (BASE_DOMAIN is a subdomain of it);
|
|
# ExternalDNS domain-filter must be the zone, not the subdomain
|
|
ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006)
|
|
GATEWAY_VIP: "10.73.0.240" # in-cluster split-horizon target (R-DNS-01/03): the L2-announced
|
|
# ingress VIP of the `shared` Gateway (first IP of the Cilium
|
|
# LB IP pool 10.73.0.240-.250, platform-config/cilium/l2-ip-pool.yaml).
|
|
# coredns-custom answers *.home.keithjackson.dev with THIS so pod->
|
|
# public-hostname traffic hairpins through the Gateway (correct SNI ->
|
|
# wildcard cert). Base default = dev VIP; a prod overlay must set the
|
|
# prod LAN VIP alongside the prod L2 pool range when hardware is built.
|
|
|
|
# --- storage (ADR-0003) ---
|
|
LONGHORN_DEFAULT_REPLICA_COUNT: "3" # default StorageClass replica count (3 on 3 nodes)
|
|
|
|
# --- observability sizing (docs/10) ---
|
|
MONITORING_PROMETHEUS_RETENTION: "7d" # dev short | prod long
|
|
MONITORING_PROMETHEUS_PVC_SIZE: "10Gi"
|
|
# SIZE-based retention cap for Prometheus (Prometheus *iB units, NOT the K8s `Gi` of the PVC knob).
|
|
# MUST stay below MONITORING_PROMETHEUS_PVC_SIZE — leaves headroom for the WAL + head + compaction
|
|
# scratch (retentionSize counts only persisted blocks). Prevents a busy TSDB from filling /prometheus
|
|
# (a full disk wedges all WAL commits and makes every absent()/==0 alert fire falsely). Base 10Gi PVC.
|
|
MONITORING_PROMETHEUS_RETENTION_SIZE: "8GiB"
|
|
MONITORING_LOKI_PVC_SIZE: "10Gi"
|
|
|
|
# --- admission enforcement (Kyverno, ADR-0011). Base defaults are the STRICT/prod posture;
|
|
# the dev overlay softens them for bootstrap (Audit / Ignore / 1) so a cold cluster can
|
|
# come up before policies are proven and admission is HA. ---
|
|
KYVERNO_VALIDATION_ACTION: "Enforce" # dev: Audit (observe before enforcing)
|
|
KYVERNO_FORCE_FAILURE_POLICY_IGNORE: "false" # dev: "true" (chart forceFailurePolicyIgnore = fail-open)
|
|
KYVERNO_ADMISSION_REPLICAS: "3" # dev: 1 (no HA needed to iterate)
|
|
|
|
# --- egress / VPN (Phase 3, ADR-0015) — node-level host OpenVPN tunnel steered by Cilium Egress
|
|
# Gateway (platform/vpn-egress/egress-steering.yaml). No in-cluster gateway pod. ---
|
|
PIA_REGION: "SE Stockholm" # PF-capable PIA region (Sweden). Under ADR-0015 the tunnel is
|
|
# host-side (Ansible vault / the node's .ovpn per docs/07), so this
|
|
# is now a reference default only — retained, not substituted into
|
|
# any manifest. ("SE Stockholm" is PF-capable; NOT "SE Streaming
|
|
# Optimized" / a US region, which lack PF.)
|
|
CLUSTER_POD_CIDR: "10.0.0.0/16" # Documents the Cilium cluster-pool pod CIDR that the egress-steering
|
|
# CiliumEgressGatewayPolicy carves out of the tunnel via excludedCIDRs
|
|
# (kept on the LOCAL datapath, never SNAT'd out tun0). The policy
|
|
# currently excludes the whole /8 RFC1918 range; this /16 records the
|
|
# actual pod allocations (Cilium assigns /24 per node from 10.0.0.0 up).
|
|
|
|
# --- identity / auth (ADR-0014) ---
|
|
KEYCLOAK_REPLICAS: "1" # dev 1 | prod 2 (HA)
|
|
OAUTH2_PROXY_REPLICAS: "1" # dev 1 | prod 2 (HA)
|
|
|
|
# --- backup / DR (ADR-0003, R-STATE-03) ---
|
|
VELERO_BACKUP_SCHEDULE: "0 3 * * *" # cron; overlays may differ
|
|
VELERO_RETENTION: "168h" # dev short | prod long
|
|
|
|
# --- placement ---
|
|
PLATFORM_NODE_SELECTOR_KEY: "kubernetes.io/os" # base no-op selector; prod may pin roles
|
|
PLATFORM_NODE_SELECTOR_VALUE: "linux"
|