home-network-k3s-impl/platform/monitoring/pvc-capacity-alerts.yaml

---
# Cluster-wide PersistentVolume capacity alerts (docs/10, R-OBS-04).
#
# WHY THIS FILE EXISTS, AND WHY IT IS NOT IN alerts.yaml. platform/monitoring/alerts.yaml has exactly
# ONE PVC-fill rule, `MonitoringVolumeFillingUp`, and it is scoped `namespace="monitoring"`. Verified
# by grep 2026-09-06: it is the repo's ONLY use of `kubelet_volume_stats_*`, so before this file NO
# database PVC — and in fact no PVC outside the monitoring namespace — was covered by any volume-fill
# alert at all. That gap is half of the 2026-09-06 CNPG archiving incident (see platform/cnpg/
# alerts.yaml): keycloak-db's 5 GiB PVC was accumulating unrecycled WAL toward a disk-full PANIC on
# the SSO database with nothing watching it.
#
# NOTE, because it changes what "covered" means: `platform-health.longhorn`'s
# `LonghornVolumeCapacityHigh` looked like it might have covered these PVCs. It never did, for two
# different reasons in sequence, and it is now GONE:
#   * 2026-09-06 — the longhorn_* series were ABSENT from Prometheus on both clusters (verified
#     read-only: `count(longhorn_volume_capacity_bytes)` empty on dev and prod), so the whole rule
#     group was inert. That was a missing Longhorn ServiceMonitor, fixed 2026-09-13 in
#     platform/longhorn/helmrelease.yaml.
#   * 2026-09-13 — with the series flowing, the rule turned out to measure the WRONG THING. It
#     divided `longhorn_volume_actual_size_bytes` (blocks ever allocated to the replica: live head
#     plus pinned snapshots, never returned without discard/TRIM) by the volume's nominal size, and
#     called the result "% used". On the Prometheus TSDB volume that read 0.9731 while the kubelet
#     read the same filesystem 20.8% used — by design and permanently, because `retentionSize:
#     45GiB` targets 90% of a 50Gi volume. It was DELETED rather than re-thresholded; see the
#     comment block in platform/monitoring/alerts.yaml. The allocation question it half-asked now
#     lives at the disk level as LonghornDiskAllocationHigh / LonghornDiskUnschedulable.
# So the rules below are not a stopgap for an inert rule group — they are the ONLY correct answer to
# "is this PVC's filesystem filling?", and they deliberately key on `kubelet_volume_stats_*`.
# MEASURED on prod 2026-09-13: of 24 Longhorn volumes, 23 appear in
# `kubelet_volume_stats_inodes > 0` and so fall in this file's scope or in
# MonitoringVolumeFillingUp's; the 24th is unifi/unifi-rootdisk, volumeMode Block, which has no
# filesystem for anyone to measure (see the inodes guard below).
#
# The rules live in a NEW FILE rather than in alerts.yaml on purpose: alerts.yaml is under concurrent
# edit, and a second file needs no coordination. Wiring is identical either way — PrometheusRule is a
# CR of the monitoring.coreos.com CRD installed by kube-prometheus-stack, so it is applied by the
# `platform-config` Flux Kustomization (dependsOn: platform), NOT the controllers pass (F-MON1), and
# `release: kube-prometheus-stack` is what makes the operator's Prometheus select it. It IS referenced
# from platform-config/kustomization.yaml.
#
# ------------------------------------------------------------------------------------------------
# HOW DOUBLE-FIRING WITH MonitoringVolumeFillingUp IS AVOIDED — by exact partition, not by luck.
#
# Two facts, both established live rather than assumed:
#
#  1. `kubelet_volume_stats_*` is published by TWO scrape jobs on these clusters, and they carry
#     DIFFERENT labels for the same PVC:
#        job="kubelet"    -> namespace=<the PVC's real namespace>, node=<node>
#        job="apiserver"  -> namespace="default", exported_namespace=<the PVC's real namespace>
#     Both jobs report the same 18 PVCs. `MonitoringVolumeFillingUp{namespace="monitoring"}` therefore
#     only ever matches the job="kubelet" copy (the apiserver copy says namespace="default"). Every
#     rule below pins `job="kubelet"` too, so both rule sets range over the SAME series universe and
#     `namespace!="monitoring"` is an EXACT complement of `namespace="monitoring"`. Without the job
#     pin, `namespace!="monitoring"` would also match the apiserver copy of the monitoring PVCs (they
#     say namespace="default") and this file WOULD double-fire on them — and would report the wrong
#     namespace while doing it.
#
#  2. Every rule below is scoped `namespace!="monitoring"`, including the critical one. So no PVC is
#     covered twice and no PVC is uncovered. The deliberate consequence: monitoring's own PVCs keep
#     only the existing 15% warning and get no critical tier. Raising one belongs in alerts.yaml,
#     which this file does not touch — recorded here as an explicit hand-off, not a silent gap.
#
# ------------------------------------------------------------------------------------------------
# WHY `and kubelet_volume_stats_inodes > 0` IS ON EVERY RULE — this guard is load-bearing.
#
# Block-mode PVCs (volumeMode: Block — the KubeVirt VM root disks) have no filesystem for the kubelet
# to stat, so it reports available=0, used=0, inodes=0 against a real capacity. `available/capacity`
# is therefore exactly 0 FOREVER. Verified live 2026-09-06: `unifi/unifi-rootdisk` (40 GiB, Block, on
# both clusters) reads 0.0. Without this guard, the warning AND the critical rule below would have
# fired permanently on it from the moment they were applied — precisely the permanently-firing-critical
# alert fatigue this repo has already been burned by (see the VPNTunnelDown post-mortem in
# alerts.yaml). `inodes > 0` is the "is there a real filesystem here?" test: any ext4/xfs volume has
# a non-zero inode total; a raw block device reports 0. Proven both ways on both clusters — with the
# guard the rules return the 15 filesystem PVCs outside monitoring and exclude unifi-rootdisk; without
# it, unifi-rootdisk is the only match at every threshold.
#
# LIVE STATE AT AUTHORING (read-only, both clusters, 2026-09-06): every rule below returns EMPTY on
# dev and on prod. Nothing here fires on day one; these are lead-time alarms, not a description of
# today. Fullest filesystem PVC outside monitoring: prod keycloak-db-1 at 93% free, dev keycloak-db-1
# at 88% free.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: platform-health-pvc-capacity
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
    platform-health: "true"
spec:
  groups:
    - name: platform-health.pvc-capacity
      rules:
        # Level alarm, warning tier. Thresholds deliberately IDENTICAL to MonitoringVolumeFillingUp
        # (<15% free, 15m) so the two halves of the partition behave the same way; only the namespace
        # scope differs.
        - alert: PersistentVolumeFillingUp
          expr: |
            (
              kubelet_volume_stats_available_bytes{job="kubelet", namespace!="monitoring"}
                / kubelet_volume_stats_capacity_bytes{job="kubelet", namespace!="monitoring"}
            ) < 0.15
            and kubelet_volume_stats_inodes{job="kubelet"} > 0
          for: 15m
          labels:
            severity: warning
          annotations:
            summary: "PVC filling up: {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }}"
            description: "PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} on node {{ $labels.node }} is under 15% free for 15m. If this is a CloudNativePG data volume (*-db-N), check WAL archiving FIRST — a failing archive_command stops Postgres recycling WAL and fills the volume (see platform/cnpg/alerts.yaml); deleting WAL by hand destroys the database."
            runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
        # Level alarm, critical tier. This is the ONLY critical in this change: it fires only when a
        # volume is genuinely about to run out, which for a Postgres data volume means an imminent
        # disk-full PANIC. Nothing on either cluster is anywhere near 5% today, so it does not add a
        # permanently-firing critical. There is no counterpart for this tier anywhere else in the
        # repo, so it cannot duplicate one.
        - alert: PersistentVolumeAlmostFull
          expr: |
            (
              kubelet_volume_stats_available_bytes{job="kubelet", namespace!="monitoring"}
                / kubelet_volume_stats_capacity_bytes{job="kubelet", namespace!="monitoring"}
            ) < 0.05
            and kubelet_volume_stats_inodes{job="kubelet"} > 0
          for: 15m
          labels:
            severity: critical
          annotations:
            summary: "PVC almost full: {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }}"
            description: "PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} on node {{ $labels.node }} has under 5% free for 15m — the workload is about to fail writes. A CloudNativePG data volume at this point will PANIC the postmaster and take the database down."
            runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
        # TREND alarm — the lead-time rule, and the point of the exercise. A level threshold is useless
        # for a slow leak on a mostly-empty volume: prod keycloak-db sat at 93% FREE while leaking
        # ~6.8 MiB/h of unrecyclable WAL, so <15% was ~29 days away and would have arrived as a
        # surprise. This rule instead asks "is this volume on course to hit zero?".
        #
        # 24h fit / 7d horizon: the 24h window is long enough to average out a daily backup or log
        # rotation (a 6h window, as the kubernetes-mixin uses, chases those and flaps), and a 7d
        # horizon gives a full week to react. `for: 1h` requires the projection to stay negative for an
        # hour, so a single burst of writes cannot notify on its own.
        #
        # The `< 0.50` gate is what keeps this quiet: a volume with more than half its space free is
        # not "about to fill" in any actionable sense, and predict_linear on a nearly-empty volume is
        # dominated by noise. It costs some lead time on the WAL case — the CNPG rules in
        # platform/cnpg/alerts.yaml are the day-one signal there, and this is the generic backstop for
        # every OTHER PVC in the cluster, which is what was missing entirely.
        #
        # NOTE the monitoring namespace is excluded here too (partition rule above). That also
        # side-steps predict_linear's classic false positive: a Prometheus TSDB filling toward its
        # retention high-water mark looks exactly like an unbounded leak until it plateaus.
        - alert: PersistentVolumeFillingUpPredicted
          expr: |
            (
              kubelet_volume_stats_available_bytes{job="kubelet", namespace!="monitoring"}
                / kubelet_volume_stats_capacity_bytes{job="kubelet", namespace!="monitoring"}
            ) < 0.50
            and predict_linear(kubelet_volume_stats_available_bytes{job="kubelet", namespace!="monitoring"}[24h], 7 * 24 * 3600) < 0
            and kubelet_volume_stats_inodes{job="kubelet"} > 0
          for: 1h
          labels:
            severity: warning
          annotations:
            summary: "PVC predicted to fill within 7 days: {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }}"
            description: "Based on the last 24h of growth, PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} on node {{ $labels.node }} is on course to run out of space within 7 days. Act now rather than at the 15%/5% thresholds: resize the PVC, prune data, or fix whatever is growing without bound (for a *-db-N volume that is usually failing WAL archiving — see platform/cnpg/alerts.yaml)."
            runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"