home-network-k3s-impl/runbooks/monitoring-capacity.md

monitoring-capacity — The monitoring stack's own PVC is filling up

Alert: MonitoringVolumeFillingUp (warning, 15m) File: platform/monitoring/alerts.yaml, group platform-health.monitoring-self Expression: kubelet_volume_stats_available_bytes{namespace="monitoring"} / kubelet_volume_stats_capacity_bytes{namespace="monitoring"} < 0.15

What this means here

"Don't let the monitor blind itself." Prometheus or Loki is running out of disk. If it fills, you lose the ability to see every other alert in this set — which is why monitoring gets its own rule instead of sharing /runbooks/pvc-capacity (the two files partition the namespace space between them; the thresholds are deliberately identical, 15% free for 15m).

Live shape on prod:

• prometheus-...-prometheus-0 PVC 50Gi, retention: 30d, retentionSize: 45GiB
• storage-loki-0 PVC 50Gi
• both on the longhorn class, 3 replicas

retentionSize: 45GiB against a 50Gi PVC is intentional: Prometheus is supposed to plateau near the high-water mark and then stay there. So a Prometheus TSDB approaching 45 GiB is normal, not a leak. That is also why the predictive rule in pvc-capacity-alerts.yaml deliberately excludes the monitoring namespace: a TSDB filling toward its retention limit looks exactly like an unbounded leak to predict_linear until it plateaus.

So the real question is which PVC and whether it should be plateauing: read the persistentvolumeclaim label first. Loki has no size cap analogous to retentionSize, so a log volume growing past 85% is the more likely genuine problem of the two.

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two dev.sh invocations at once; they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git (git push origin main is the deploy). Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring get pvc; echo; \
   kubectl --context k3s-prod -n monitoring get prometheus -o jsonpath="{.items[0].spec.retention} {.items[0].spec.retentionSize}{\"\n\"}"'

Actual on-disk usage from inside the pods:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- df -h /prometheus; \
   kubectl --context k3s-prod -n monitoring exec loki-0 -- df -h /var/loki'

Is Prometheus at its retentionSize (normal) or past it (a problem)?

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   for q in "prometheus_tsdb_storage_blocks_bytes" "prometheus_tsdb_head_series" "rate(prometheus_tsdb_compactions_failed_total[1h])"; do \
     printf "%s " "$q"; curl -s -G --data-urlencode "query=$q" localhost:19090/api/v1/query | python3 -c "import json,sys;print([x[\"value\"][1] for x in json.load(sys.stdin)[\"data\"][\"result\"]])"; done'

Series growth is the usual driver — find the biggest contributors:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   curl -s "localhost:19090/api/v1/status/tsdb" | python3 -m json.tool | head -40'

Likely causes, most likely first

1. Prometheus has reached its retention high-water mark (expected)

retentionSize: 45GiB on a 50Gi PVC leaves 10% headroom, which is close to the 15% trigger. If prometheus_tsdb_storage_blocks_bytes is near 45 GiB and steady, this is the design working and the correct response is either to accept it or to raise both the PVC and retentionSize together in platform/monitoring/kube-prometheus-stack.yaml. Raising retention without raising the PVC is how you get a genuinely full disk.

2. Loki log volume growth

Loki has no retentionSize equivalent configured. A noisy workload (a crashlooping pod, a chatty reconciler) can grow it quickly. Remediation: find the noisy stream in Grafana → Explore, fix the source, and set/lower Loki retention in git.

3. Series cardinality grew after a chart or workload change

serviceMonitorSelectorNilUsesHelmValues and podMonitorSelectorNilUsesHelmValues are both false here, which means Prometheus selects every ServiceMonitor/PodMonitor in the cluster regardless of labels. That is convenient (no additionalLabels needed anywhere) and it also means a new component's monitor is picked up automatically, cardinality and all. Check /api/v1/status/tsdb for the top label pairs.

4. Compaction is failing

rate(prometheus_tsdb_compactions_failed_total[1h]) > 0 means blocks are not being merged or expired. Usually an I/O problem underneath — cross-read /runbooks/node-io.

5. The volume is on a node that is itself short of space

/runbooks/longhorn-capacity and /runbooks/node-disk.

When to escalate or stop

• Do not delete Prometheus TSDB blocks or Loki chunks by hand. Lower retention and let the component expire them; hand-deleting blocks can leave the TSDB unopenable.
• Do not raise retention or retentionSize without raising the PVC in the same change.
• If Prometheus is already full and crashlooping, you are blind: check the alerts you care about directly against the CRs (clusters.postgresql.cnpg.io, volumes.longhorn.io, kubectl get nodes) rather than waiting for alerts that cannot be evaluated.
• Escalate for PVC expansion — it needs node headroom, and this cluster has been memory- and disk-constrained before (Prometheus was already raised 2 → 3Gi of RAM after the kps 65→88 upgrade).

---

Source of truth: runbooks/monitoring-capacity.md. Index: /runbooks/ Records: platform/monitoring/pvc-capacity-alerts.yaml (why monitoring is excluded from the predictive rule); DECISIONS-LOG.md (Grafana OOM / Prometheus 2→3Gi during kps 65→88).