home-network-k3s-impl/runbooks/pvc-capacity.md

# pvc-capacity — A PersistentVolumeClaim is filling up, nearly full, or predicted to fill

**Alerts:** `PersistentVolumeFillingUp` (warning, 15m), `PersistentVolumeAlmostFull` (critical, 15m),
`PersistentVolumeFillingUpPredicted` (warning, 1h)
**File:** `platform/monitoring/pvc-capacity-alerts.yaml`, group `platform-health.pvc-capacity`
**Expressions:** `< 0.15` free / `< 0.05` free / `< 0.50` free **and**
`predict_linear(kubelet_volume_stats_available_bytes[24h], 7*24*3600) < 0`, each `and
kubelet_volume_stats_inodes{job="kubelet"} > 0`

## What this means here

These are the PVC fill alerts for everything **outside** the `monitoring` namespace
(`/runbooks/monitoring-capacity` owns that half). Before this file existed, **no PVC outside
`monitoring` had any fill alert at all**. Unlike the Longhorn rule group, these work: they are built
on `kubelet_volume_stats_*`, which is live (56 series on prod).

Three design facts you need in order to read the labels correctly:

- **Every rule pins `job="kubelet"`.** `kubelet_volume_stats_*` is published by *two* jobs with
  different label shapes: `job="kubelet"` carries the PVC's real namespace, while `job="apiserver"`
  reports `namespace="default"` with the real one in `exported_namespace`. Without the pin,
  `namespace!="monitoring"` double-fires on monitoring's own PVCs and names the wrong namespace.
- **Every rule carries `and kubelet_volume_stats_inodes > 0`.** Block-mode PVCs — KubeVirt VM disks
  on `longhorn-vm` — report available=0, used=0, inodes=0 against a real capacity. Without the guard,
  `unifi/unifi-rootdisk` (40 GiB Block) would fire both warning **and** critical permanently. If you
  are ever tempted to remove that clause, this is why it is there.
- **The predicted rule is the point of the exercise.** A level threshold is useless for a slow leak on
  a mostly-empty volume: prod's `keycloak-db` sat at **93% free** while leaking ~6.8 MiB/h of
  unrecyclable WAL, so `<15%` was ~29 days away and would have arrived as a surprise. The `< 0.50`
  gate keeps it quiet on volumes that are not plausibly about to fill, and the 24h fit / 7d horizon is
  chosen so a daily backup or log rotation does not make it flap.

**If the PVC name matches `*-db-N`, read `/runbooks/cnpg-archiving` before doing anything else.**
That is by far the most likely cause on this cluster and the one where the obvious action is fatal.
## Running the commands in this runbook

`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.

## First commands

Which PVC, how full, on which node:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   curl -s -G --data-urlencode "query=topk(10, 1 - kubelet_volume_stats_available_bytes{job=\"kubelet\"} / kubelet_volume_stats_capacity_bytes{job=\"kubelet\"})" localhost:19090/api/v1/query | python3 -m json.tool | grep -E "persistentvolumeclaim|namespace|\"1\"" '
```

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get pvc -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SIZE:.spec.resources.requests.storage,SC:.spec.storageClassName,STATUS:.status.phase'
```

Is it a CNPG data volume? (Fully-qualified `clusters.postgresql.cnpg.io` — the short name `cluster` is
ambiguous on clusters carrying more than one `Cluster` kind.)

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A'
```

If yes, the one question that decides everything — is WAL archiving working?

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,LASTBACKUP:.status.conditions[?\(@.type==\"LastBackupSucceeded\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint'
```

What is actually consuming the space, from inside the pod:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak exec keycloak-db-2 -c postgres -- bash -c "df -h /var/lib/postgresql/data; du -sh /var/lib/postgresql/data/pgdata/pg_wal; ls /var/lib/postgresql/data/pgdata/pg_wal/archive_status | wc -l"'
```

## Likely causes, most likely first

### 1. A CNPG database is not recycling WAL
The dominant cause. With `archive_mode=on` and a failing `archive_command`, Postgres **refuses to
recycle** unarchived WAL segments, so `pg_wal` grows until the data PVC is full and the postmaster
**PANICs** — taking the database, and for `keycloak-db` all of SSO, down. `pg_wal` shares the data
PVC on every Cluster here (no `walStorage:` is configured), which is why a *backup* fault presents as
a *capacity* alert. This actually happened: prod ran from bootstrap to 2026-09-07 with
`endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`, all three Clusters at `ContinuousArchiving=False` with
`barman-cloud-wal-archive: exit status 4`, unnoticed for 38+ hours.
**Remediation: fix archiving (`/runbooks/cnpg-archiving`). Do NOT delete WAL by hand — that destroys
the database. Do NOT expand the PVC and call it done; that buys days.**

### 2. Log or cache growth in an app-tier workload
Nextcloud, the *arr apps, Gitea. Prune inside the app, or expand the PVC in git (every Longhorn class
has `allowVolumeExpansion: true`).

### 3. Media on the wrong storage class
Media is meant to be static NFS PVs from `192.168.2.102:/mnt/Storage/plex` (8.32 TiB measured), not
Longhorn. A media volume on `longhorn` is a misconfiguration; fix the class, do not grow the disk.

### 4. A Velero or CDI staging write
Transient. CDI uses `longhorn-scratch` deliberately (single replica) because the 3-replica default
failed the UniFi 60Gi import on insufficient storage.

### 5. The predicted rule caught a genuine slow leak
This is the rule working as designed and giving you a week. Find what is growing before the level
thresholds arrive.

## When to escalate or stop

- **`PersistentVolumeAlmostFull` on a `*-db-N` volume is an imminent database outage.** Get the
  operator. A Postgres data volume at under 5% free will PANIC the postmaster.
- **Never delete WAL segments, and never delete anything under a Postgres data directory.**
- **Do not delete the PVC** to reclaim space.
- If `unifi-rootdisk` or another Block-mode volume appears in these alerts, the inodes guard has been
  removed or broken — that is a monitoring bug, not a storage incident.
- Escalate for PVC expansion beyond what the node can hold (see `/runbooks/longhorn-capacity`).

---
Source of truth: `runbooks/pvc-capacity.md`. Index: `/runbooks/`
Records: `platform/monitoring/pvc-capacity-alerts.yaml` (the two-job and inodes findings);
`platform/cnpg/alerts.yaml` header (the 2026-09-06 archiving incident and the measured fill rate).