Files
home-network-k3s-impl/clusters/dev/kustomization.yaml
T
jakman85andClaude Opus 4.8 eb31d2ef58 fix(monitoring): cap Prometheus TSDB by size to stop false absent()/==0 alerts
The 7 falsely-firing platform alerts (KeycloakDown, KeycloakDatabaseDown,
KubeAPIDown, KubeletDown, VPNTunnelDown, KubeVirtControlPlaneDown,
CDINoDefaultStorageClass) share ONE root cause, not per-rule label mismatches:
the Prometheus PVC is 100% full ("no space left on device" on every WAL commit).

Evidence from the live dev cluster:
- /prometheus is 4.8G/5Gi, 0 bytes free; every scrape logs "Scrape commit
  failed ... write to WAL: no space left on device". Scrapes still return 200
  (targets show healthy, lastScrape=now) but samples are never stored, so the
  TSDB head froze at 2026-08-28 13:45 UTC while the pod clock reads ~19:33 UTC.
- Instant queries at "now" therefore return an EMPTY vector, so every
  absent()/==0 platform rule trips. Evaluated at the last-good-sample time each
  expr is correctly FALSE (keycloak-0/keycloak-db-1 ready=1, apiserver+kubelet
  up=1 x3, tun0 series present, kubevirt control-plane expr empty, CDI default
  present) — the rules are correct; the monitor was simply blind.

`retention` alone prunes only by AGE: this cluster is 3d2h old with 3d dev
retention, so time-pruning has NEVER run and 3d of block churn (~1.2Gi/day) plus
a WAL that ballooned past 1Gi overran the 5Gi disk. Fix adds a size-based
retention cap (new MONITORING_PROMETHEUS_RETENTION_SIZE knob) kept below each
env's PVC so Prometheus drops oldest blocks before the disk fills, and bumps the
dev PVC 5Gi->8Gi for real WAL/head/compaction headroom.

- platform/monitoring/kube-prometheus-stack.yaml: retentionSize: ${knob}
- clusters/base/cluster-vars.yaml: knob default 8GiB (base 10Gi PVC)
- clusters/dev/kustomization.yaml: PVC 5Gi->8Gi, retentionSize 5GiB
- clusters/prod/kustomization.yaml: retentionSize 45GiB (50Gi PVC)
- platform/monitoring/README.md: knob table + rationale

Deploy note (PM): the PVC is already 100% full, so Prometheus cannot self-prune
until it can write. The 5Gi->8Gi bump must land as a Longhorn online expansion
(operator PVC expansion / manual patch); once free space returns, retentionSize
keeps it healthy. Watchdog left untouched.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-28 14:38:12 -05:00

176 lines
7.3 KiB
YAML

---
# clusters/dev — the ACTIVE build target overlay.
#
# `flux bootstrap ... --path=clusters/dev` points the dev cluster's Flux here. It reuses the base
# reconciliation graph and patches the shared cluster-vars ConfigMap with dev-scoped values (R-ENV-02).
# It also adds a self-managing ROOT Kustomization (sync.yaml) so the whole graph reconciles from git
# on every push — no more manual `kubectl apply -k` for cluster-vars/overlay changes (F-OPS1).
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../base
- sync.yaml # self-managing root Kustomization → ./clusters/dev
patches:
# --- SOPS decryption + bootstrap source (PM-inlined from bootstrap/flux/patches/, 2026-08-16) ---
# Inlined (not cross-dir file refs) to stay within kustomize's default load restrictions.
# Realises the commented decryption TODOs in clusters/base and sets the real source URL.
# The `sops-age` Secret is created out-of-band (bootstrap/sops/install-age-key.sh) — name only.
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: platform
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
# SOPS decryption on EACH app wave (the app tier is now apps-wave-0..N, not a single `apps`
# Kustomization). Same stanza the retired `apps` Kustomization received — one per wave, by
# name. Add a patch here whenever `render.py --emit-waves` introduces a new deepest wave
# (DESIGN-waves.md cutover note). The `sops-age` Secret is created out-of-band (name only).
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: apps-wave-0
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: apps-wave-1
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: apps-wave-2
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
# oauth2-proxy moved to its own post-platform Flux Kustomization (clusters/base/oauth2-proxy.yaml).
# It carries oauth2-proxy-credentials.sops.yaml, previously decrypted via the `platform`
# Kustomization; now that this tier owns the encrypted Secret it needs its own decryption stanza.
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: oauth2-proxy
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
# headlamp moved to its own post-platform Flux Kustomization (clusters/base/headlamp.yaml).
# It carries headlamp-oidc-credentials.sops.yaml, previously decrypted via the `platform`
# Kustomization; now that this tier owns the encrypted Secret it needs its own decryption stanza.
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: headlamp
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
- patch: |-
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: flux-system
namespace: flux-system
spec:
# Phase-6 cutover done: source is now the in-cluster Gitea; bootstrap host-Gitea retained
# stopped as recovery. PUBLIC repo => Flux needs no creds; Flux resolves the host in-cluster
# via coredns -> Gateway VIP, and the LE-prod wildcard cert is publicly trusted so NO
# secretRef/caCert is needed. Recovery: restart bootstrap Gitea (192.168.2.103:3000) and
# revert this url (docs/05).
url: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
ref:
branch: main
# --- dev-scoped cluster-vars (R-ENV-02) ---
- target:
kind: ConfigMap
name: cluster-vars
patch: |-
- op: replace
path: /data/CLUSTER_ENV
value: "dev"
# DNS-01 via LE PRODUCTION: in-cluster OIDC clients (oauth2-proxy, Grafana) must trust the
# wildcard cert for SSO to work; the staging root isn't in any trust store. Prod issuer is
# registered + Ready; one wildcard cert is well within LE rate limits — ADR-0006, R-TLS-01/03.
- op: replace
path: /data/ACME_CLUSTER_ISSUER
value: "letsencrypt-prod"
# Split-horizon DNS target: the Cilium-L2 VIP of the `shared` Gateway on the Proxmox LAN
# (base default 10.73.0.240 was the old libvirt VIP). coredns-custom answers
# *.home.keithjackson.dev with this so in-cluster OIDC clients reach Keycloak via the Gateway.
- op: replace
path: /data/GATEWAY_VIP
value: "192.168.1.53"
# dev mirrors prod's 3-node topology, so the replicated tier keeps 3 replicas.
- op: replace
path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
value: "3"
# smaller retention / volumes to keep the dev workstation footprint light.
- op: replace
path: /data/MONITORING_PROMETHEUS_RETENTION
value: "3d"
# 5Gi was too tight: 3d of this cluster's series volume (~1.2Gi/day of blocks) plus a WAL that
# ballooned past 1Gi overran the disk, wedging WAL commits and firing every absent()/==0 alert
# falsely. Bumped to 8Gi (still lighter than base 10Gi) so the size-cap below has real headroom.
- op: replace
path: /data/MONITORING_PROMETHEUS_PVC_SIZE
value: "8Gi"
# Size-cap kept well below the 8Gi PVC (headroom for WAL + head + compaction scratch) so
# Prometheus prunes oldest blocks by size and never fills the disk on this young cluster.
- op: replace
path: /data/MONITORING_PROMETHEUS_RETENTION_SIZE
value: "5GiB"
- op: replace
path: /data/MONITORING_LOKI_PVC_SIZE
value: "5Gi"
# single-replica control-plane addons in dev (no HA needed to iterate).
- op: replace
path: /data/KEYCLOAK_REPLICAS
value: "1"
- op: replace
path: /data/OAUTH2_PROXY_REPLICAS
value: "1"
- op: replace
path: /data/VELERO_RETENTION
value: "72h"
# dev-softened Kyverno for cold-start (base defaults Enforce/Fail/3 = prod posture).
# Not credential-driven — bootstrap-safety knobs so a cold cluster comes up before
# admission is proven/HA; flip dev to the strict posture once green.
- op: replace
path: /data/KYVERNO_VALIDATION_ACTION
value: "Audit"
- op: replace
path: /data/KYVERNO_FORCE_FAILURE_POLICY_IGNORE
value: "true"
# 2 (not base-default 1): the webhook is fail-closed, so a single-pod restart 502s every
# admission-gated write cluster-wide. Two replicas give a fail-closed webhook availability
# headroom during a restart/upgrade while staying dev-modest (prod runs 3).
- op: replace
path: /data/KYVERNO_ADMISSION_REPLICAS
value: "2"