home-network-k3s-impl/runbooks/pvc-capacity.md

pvc-capacity — A PersistentVolumeClaim is filling up, nearly full, or predicted to fill

Alerts: PersistentVolumeFillingUp (warning, 15m), PersistentVolumeAlmostFull (critical, 15m), PersistentVolumeFillingUpPredicted (warning, 1h) File: platform/monitoring/pvc-capacity-alerts.yaml, group platform-health.pvc-capacity Expressions: < 0.15 free / < 0.05 free / < 0.50 free and predict_linear(kubelet_volume_stats_available_bytes[24h], 7*24*3600) < 0, each and kubelet_volume_stats_inodes{job="kubelet"} > 0

What this means here

These are the PVC fill alerts for everything outside the monitoring namespace (/runbooks/monitoring-capacity owns that half). Before this file existed, no PVC outside monitoring had any fill alert at all. Unlike the Longhorn rule group, these work: they are built on kubelet_volume_stats_*, which is live (56 series on prod).

Three design facts you need in order to read the labels correctly:

• Every rule pins job="kubelet". kubelet_volume_stats_* is published by two jobs with different label shapes: job="kubelet" carries the PVC's real namespace, while job="apiserver" reports namespace="default" with the real one in exported_namespace. Without the pin, namespace!="monitoring" double-fires on monitoring's own PVCs and names the wrong namespace.
• Every rule carries and kubelet_volume_stats_inodes > 0. Block-mode PVCs — KubeVirt VM disks on longhorn-vm — report available=0, used=0, inodes=0 against a real capacity. Without the guard, unifi/unifi-rootdisk (40 GiB Block) would fire both warning and critical permanently. If you are ever tempted to remove that clause, this is why it is there.
• The predicted rule is the point of the exercise. A level threshold is useless for a slow leak on a mostly-empty volume: prod's keycloak-db sat at 93% free while leaking ~6.8 MiB/h of unrecyclable WAL, so <15% was ~29 days away and would have arrived as a surprise. The < 0.50 gate keeps it quiet on volumes that are not plausibly about to fill, and the 24h fit / 7d horizon is chosen so a daily backup or log rotation does not make it flap.

If the PVC name matches *-db-N, read /runbooks/cnpg-archiving before doing anything else. That is by far the most likely cause on this cluster and the one where the obvious action is fatal.

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two dev.sh invocations at once; they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git (git push origin main is the deploy). Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

Which PVC, how full, on which node:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   curl -s -G --data-urlencode "query=topk(10, 1 - kubelet_volume_stats_available_bytes{job=\"kubelet\"} / kubelet_volume_stats_capacity_bytes{job=\"kubelet\"})" localhost:19090/api/v1/query | python3 -m json.tool | grep -E "persistentvolumeclaim|namespace|\"1\"" '

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get pvc -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SIZE:.spec.resources.requests.storage,SC:.spec.storageClassName,STATUS:.status.phase'

Is it a CNPG data volume? (Fully-qualified clusters.postgresql.cnpg.io — the short name cluster is ambiguous on clusters carrying more than one Cluster kind.)

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A'

If yes, the one question that decides everything — is WAL archiving working?

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,LASTBACKUP:.status.conditions[?\(@.type==\"LastBackupSucceeded\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint'

What is actually consuming the space, from inside the pod:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak exec keycloak-db-2 -c postgres -- bash -c "df -h /var/lib/postgresql/data; du -sh /var/lib/postgresql/data/pgdata/pg_wal; ls /var/lib/postgresql/data/pgdata/pg_wal/archive_status | wc -l"'

Likely causes, most likely first

1. A CNPG database is not recycling WAL

The dominant cause. With archive_mode=on and a failing archive_command, Postgres refuses to recycle unarchived WAL segments, so pg_wal grows until the data PVC is full and the postmaster PANICs — taking the database, and for keycloak-db all of SSO, down. pg_wal shares the data PVC on every Cluster here (no walStorage: is configured), which is why a backup fault presents as a capacity alert. This actually happened: prod ran from bootstrap to 2026-09-07 with endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT, all three Clusters at ContinuousArchiving=False with barman-cloud-wal-archive: exit status 4, unnoticed for 38+ hours. Remediation: fix archiving (/runbooks/cnpg-archiving). Do NOT delete WAL by hand — that destroys the database. Do NOT expand the PVC and call it done; that buys days.

2. Log or cache growth in an app-tier workload

Nextcloud, the *arr apps, Gitea. Prune inside the app, or expand the PVC in git (every Longhorn class has allowVolumeExpansion: true).

3. Media on the wrong storage class

Media is meant to be static NFS PVs from 192.168.2.102:/mnt/Storage/plex (8.32 TiB measured), not Longhorn. A media volume on longhorn is a misconfiguration; fix the class, do not grow the disk.

4. A Velero or CDI staging write

Transient. CDI uses longhorn-scratch deliberately (single replica) because the 3-replica default failed the UniFi 60Gi import on insufficient storage.

5. The predicted rule caught a genuine slow leak

This is the rule working as designed and giving you a week. Find what is growing before the level thresholds arrive.

When to escalate or stop

• PersistentVolumeAlmostFull on a *-db-N volume is an imminent database outage. Get the operator. A Postgres data volume at under 5% free will PANIC the postmaster.
• Never delete WAL segments, and never delete anything under a Postgres data directory.
• Do not delete the PVC to reclaim space.
• If unifi-rootdisk or another Block-mode volume appears in these alerts, the inodes guard has been removed or broken — that is a monitoring bug, not a storage incident.
• Escalate for PVC expansion beyond what the node can hold (see /runbooks/longhorn-capacity).

---

Source of truth: runbooks/pvc-capacity.md. Index: /runbooks/ Records: platform/monitoring/pvc-capacity-alerts.yaml (the two-job and inodes findings); platform/cnpg/alerts.yaml header (the 2026-09-06 archiving incident and the measured fill rate).