home-network-k3s-impl/runbooks/monitoring-capacity.md

# monitoring-capacity — The monitoring stack's own PVC is filling up

**Alert:** `MonitoringVolumeFillingUp` (warning, 15m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.monitoring-self`
**Expression:** `kubelet_volume_stats_available_bytes{namespace="monitoring"} / kubelet_volume_stats_capacity_bytes{namespace="monitoring"} < 0.15`

## What this means here

"Don't let the monitor blind itself." Prometheus or Loki is running out of disk. If it fills, you lose
the ability to see every other alert in this set — which is why `monitoring` gets its own rule instead
of sharing `/runbooks/pvc-capacity` (the two files partition the namespace space between them; the
thresholds are deliberately identical, 15% free for 15m).

Live shape on prod:

- `prometheus-...-prometheus-0` PVC **50Gi**, `retention: 30d`, `retentionSize: 45GiB`
- `storage-loki-0` PVC **50Gi**
- both on the `longhorn` class, 3 replicas

`retentionSize: 45GiB` against a 50Gi PVC is intentional: Prometheus is *supposed* to plateau near the
high-water mark and then stay there. **So a Prometheus TSDB approaching 45 GiB is normal, not a leak.**
That is also why the predictive rule in `pvc-capacity-alerts.yaml` deliberately **excludes** the
`monitoring` namespace: a TSDB filling toward its retention limit looks exactly like an unbounded leak
to `predict_linear` until it plateaus.

So the real question is *which* PVC and *whether it should be plateauing*: read the
`persistentvolumeclaim` label first. Loki has no size cap analogous to `retentionSize`, so a log
volume growing past 85% is the more likely genuine problem of the two.
## Running the commands in this runbook

`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.

## First commands

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring get pvc; echo; \
   kubectl --context k3s-prod -n monitoring get prometheus -o jsonpath="{.items[0].spec.retention} {.items[0].spec.retentionSize}{\"\n\"}"'
```

Actual on-disk usage from inside the pods:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- df -h /prometheus; \
   kubectl --context k3s-prod -n monitoring exec loki-0 -- df -h /var/loki'
```

Is Prometheus at its retentionSize (normal) or past it (a problem)?

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   for q in "prometheus_tsdb_storage_blocks_bytes" "prometheus_tsdb_head_series" "rate(prometheus_tsdb_compactions_failed_total[1h])"; do \
     printf "%s " "$q"; curl -s -G --data-urlencode "query=$q" localhost:19090/api/v1/query | python3 -c "import json,sys;print([x[\"value\"][1] for x in json.load(sys.stdin)[\"data\"][\"result\"]])"; done'
```

Series growth is the usual driver — find the biggest contributors:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   curl -s "localhost:19090/api/v1/status/tsdb" | python3 -m json.tool | head -40'
```

## Likely causes, most likely first

### 1. Prometheus has reached its retention high-water mark (expected)
`retentionSize: 45GiB` on a 50Gi PVC leaves 10% headroom, which is close to the 15% trigger. If
`prometheus_tsdb_storage_blocks_bytes` is near 45 GiB and steady, **this is the design working** and
the correct response is either to accept it or to raise both the PVC and `retentionSize` together in
`platform/monitoring/kube-prometheus-stack.yaml`. Raising retention without raising the PVC is how you
get a genuinely full disk.

### 2. Loki log volume growth
Loki has no retentionSize equivalent configured. A noisy workload (a crashlooping pod, a chatty
reconciler) can grow it quickly. Remediation: find the noisy stream in Grafana → Explore, fix the
source, and set/lower Loki retention in git.

### 3. Series cardinality grew after a chart or workload change
`serviceMonitorSelectorNilUsesHelmValues` and `podMonitorSelectorNilUsesHelmValues` are both `false`
here, which means **Prometheus selects every ServiceMonitor/PodMonitor in the cluster regardless of
labels**. That is convenient (no `additionalLabels` needed anywhere) and it also means a new
component's monitor is picked up automatically, cardinality and all. Check
`/api/v1/status/tsdb` for the top label pairs.

### 4. Compaction is failing
`rate(prometheus_tsdb_compactions_failed_total[1h]) > 0` means blocks are not being merged or expired.
Usually an I/O problem underneath — cross-read `/runbooks/node-io`.

### 5. The volume is on a node that is itself short of space
`/runbooks/longhorn-capacity` and `/runbooks/node-disk`.

## When to escalate or stop

- **Do not delete Prometheus TSDB blocks or Loki chunks by hand.** Lower retention and let the
  component expire them; hand-deleting blocks can leave the TSDB unopenable.
- **Do not raise `retention` or `retentionSize` without raising the PVC in the same change.**
- If Prometheus is already full and crashlooping, you are blind: check the alerts you care about
  directly against the CRs (`clusters.postgresql.cnpg.io`, `volumes.longhorn.io`, `kubectl get nodes`)
  rather than waiting for alerts that cannot be evaluated.
- Escalate for PVC expansion — it needs node headroom, and this cluster has been memory- and
  disk-constrained before (Prometheus was already raised 2 → 3Gi of RAM after the kps 65→88 upgrade).

---
Source of truth: `runbooks/monitoring-capacity.md`. Index: `/runbooks/`
Records: `platform/monitoring/pvc-capacity-alerts.yaml` (why monitoring is excluded from the
predictive rule); `DECISIONS-LOG.md` (Grafana OOM / Prometheus 2→3Gi during kps 65→88).