# longhorn-capacity — A Longhorn disk is running out of room for replicas **Alerts:** `LonghornDiskAllocationHigh` (warning, 30m) and `LonghornDiskUnschedulable` (warning, 15m) **File:** `platform/monitoring/alerts.yaml`, group `platform-health.longhorn` **Expressions:** `(longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70` `longhorn_disk_status{condition="schedulable"} == 0` ## What this means here **This is a POOL question, not a PVC question.** One of the three nodes' Longhorn data disks is filling with replica files, or Longhorn has already stopped placing replicas on it. Neither alert says anything about how full any single workload's filesystem is — for that, read `/runbooks/pvc-capacity` (`PersistentVolumeFillingUp`, `PersistentVolumeAlmostFull`) or `/runbooks/monitoring-capacity` (`MonitoringVolumeFillingUp`). Those are keyed on `kubelet_volume_stats_*`, which is the filesystem's own accounting. `longhorn_disk_usage_bytes` counts bytes ALLOCATED: every replica file plus every snapshot still pinned to it. Longhorn does not return freed blocks to a replica file without discard/TRIM, and a system snapshot taken for a replica rebuild keeps its historical blocks alive, so this number only goes up until something is deleted. That is the correct question *here* — allocated bytes are what consumes the node's ~414 GiB data partition — and the wrong question per volume, which is why `LonghornVolumeCapacityHigh` was deleted on 2026-09-13 (it read 0.973 on a Prometheus TSDB volume whose filesystem was 20.8% used, by design, permanently). See the comment block in `platform/monitoring/alerts.yaml` for the measurements. **Why 0.70 and not 0.85.** `storageMinimalAvailablePercentage: 25` means Longhorn itself refuses to schedule a new replica on a disk with under 25% available — i.e. at **75% used the disk is already out of the scheduling pool**. An 85% threshold could only ever fire after that had happened. 0.70 is the last point at which there is still lead time. The companion settings are `storageReservedPercentageForDefaultDisk: 20`, `storageOverProvisioningPercentage: 200`, `defaultDataLocality: disabled`, hard node anti-affinity. Measured on prod 2026-09-13: usage 0.2254 / 0.2306 / 0.2369 on prod-1/2/3, with 268.9 GiB of nominal replica size scheduled against (414.5 GiB − 20% reserved) × 200%. So either alert firing means something changed materially, not that the platform slowly grew. The severe consequence is that a degraded volume **cannot finish rebuilding** — hard anti-affinity needs three distinct nodes with room, so one full or unschedulable disk pins a volume at 2 replicas indefinitely, and R-STATE-02's survive-a-node-loss promise is suspended for as long as that lasts. Cross-read `/runbooks/longhorn-degraded`. ### Verify the alert can fire (measured 2026-09-13) These alerts were INERT until 2026-09-13, when `metrics.serviceMonitor.enabled` was set in `platform/longhorn/helmrelease.yaml`; the longhorn_* family was simply absent from the TSDB before that. Both are now live — verified on prod the same day: ``` infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \ prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \ /bin/promtool query instant http://localhost:9090 \ 'count(longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) + count(longhorn_disk_status{condition="schedulable"})' ``` returned **6** (3 disks, both selectors resolving), while both alert expressions returned empty. If that count is ever empty again, the scrape has broken and this pair is silently back to inert — check `up{job="longhorn-backend"}` and `platform/longhorn/networkpolicy-metrics.yaml`. ## Running the commands in this runbook `kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write `backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for agents here: every durable change goes through git. **`main` is a protected branch — a direct push is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, and it says so. ## First commands Per-disk allocation and Longhorn's own scheduling verdict — the two things the alerts read: ``` infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \ prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \ /bin/promtool query instant http://localhost:9090 \ 'longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes or longhorn_disk_status' ``` Node-level allocation, from the CRs: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o yaml | grep -E "name:|storageAvailable|storageMaximum|storageScheduled|allowScheduling"' ``` Volume nominal size versus ALLOCATED size. `actualSize` is head + pinned snapshots, NOT filesystem usage — a volume can read near 100% here with a nearly empty filesystem (see the deleted-alert note above), so do not read this column as "full": ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \ -o custom-columns=NAME:.metadata.name,SPEC:.spec.size,ACTUAL:.status.actualSize,ROBUST:.status.robustness' ``` Which PVC each one is, so you know who is filling it: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get pv -o custom-columns=PV:.metadata.name,CLAIM:.spec.claimRef.namespace/.spec.claimRef.name,SIZE:.spec.capacity.storage,SC:.spec.storageClassName' ``` Which snapshots are holding the allocation (this is the measured top cause on prod — one `user_created=false` system snapshot on the Prometheus TSDB volume held 37.7 GiB on 2026-09-13): ``` infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \ prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \ /bin/promtool query instant http://localhost:9090 \ 'topk(10, longhorn_snapshot_actual_size_bytes)' ``` The filesystem view — the separate question, answered by `/runbooks/pvc-capacity`: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ curl -s -G --data-urlencode "query=topk(10, 1 - kubelet_volume_stats_available_bytes{job=\"kubelet\"} / kubelet_volume_stats_capacity_bytes{job=\"kubelet\"})" localhost:19090/api/v1/query' ``` On the node itself: ``` ssh k3s-prod-1 'df -h /data; du -xh --max-depth=1 /data/longhorn/replicas 2>/dev/null | sort -h | tail' ``` ## Likely causes, most likely first ### 0. Pinned snapshots, not live data (check this FIRST for an allocation alert) An allocation alert can fire with every filesystem in the cluster nearly empty. Run the `longhorn_snapshot_actual_size_bytes` query above before anything else. A replica rebuild — which is what a node reboot or a disk replacement forces, and prod-1's drive was replaced on 2026-09-12 — leaves a `user_created=false` system snapshot behind, and it keeps its blocks until it is purged. Prune from the Longhorn UI (Volume → Snapshots), then re-read the query; nothing else is needed if this was the cause. ### 1. A CNPG database is not recycling WAL (check this FIRST for a PVC-fill alert) The only cause here that grows without bound. `archive_mode=on` plus a failing `archive_command` means Postgres will not recycle unarchived WAL; measured on prod `keycloak-db` at ~6.8 MiB/h with `pg_wal` sharing the data PVC (no `walStorage:` is configured on any Cluster) — about 29 days from 93% free to disk-full, ending in a postmaster PANIC. Go to `/runbooks/cnpg-archiving`. **Never delete WAL by hand.** ### 2. Snapshots and orphaned replicas accumulating Longhorn snapshots are not free, and cleanup here is **manual**. `orphanAutoDeletion: true` used to sit in `platform/longhorn/helmrelease.yaml` and had been **silently inert since Longhorn 1.12** — it was renamed `orphanResourceAutoDeletion` (now a list) and the chart drops unknown `defaultSettings` keys with no error. It was **removed on 2026-09-13** and deliberately not replaced, so orphan cleanup is manual by decision, not by accident: prune orphans and stale snapshots from the Longhorn UI. Note also that **snapshot purge and clone now run one at a time per node** (`snapshotHeavyTaskConcurrentLimit: 1`, declared 2026-09-13 to keep Longhorn from starving etcd's WAL fsync). Reclaim after a rebuild storm is therefore *serial and slower by design*. If allocation is climbing and purge is the bottleneck, the operator may raise the Setting for the duration and watch `NodeIOPressureHigh` — see `runbooks/longhorn-degraded.md` for the tradeoff as recorded. ### 3. A workload genuinely grew Plex/Sonarr/Radarr **media** does not live on Longhorn — it is static NFS from `192.168.2.102:/mnt/Storage/plex`, via the `bulk` tier. If a media volume is on `longhorn`, that is a misconfiguration worth fixing, not a capacity problem: the `bulk` StorageClass is a stub (`provisioner: nfs.csi.k8s.io`, `server:/share: REPLACE-ME`) and provisions nothing, so real bulk storage is static NFS PVs. Otherwise expand the PVC — all Longhorn classes have `allowVolumeExpansion: true`; change the size in git and push. ### 4. Replicas piled onto one node `replicaAutoBalance: best-effort` should prevent this, and hard anti-affinity bounds it to one replica per volume per node. If one node is much fuller than the others, check whether a rebuild placed replicas there while another node was down. ### 5. The cluster has outgrown its disks Prod's ceiling is ~829 GiB across three nodes. `docs/plans/longhorn-ssd-tiering.md` is the planned answer (a ~51 GB hot tier: Prometheus, CNPG, Loki, Gitea). Operator planning work. ## When to escalate or stop - **A disk over 70% while any volume is degraded is urgent**, because the rebuild cannot complete — and at 75% Longhorn takes the disk out of the scheduling pool entirely (`LonghornDiskUnschedulable`). Free space or restore the other node before anything else. - **Never delete files under `/data/longhorn`** or inside a Postgres `pg_wal`. Both are irreversible. - **Do not expand a PVC as a reflex** if the cause is unrecycled WAL — you are buying days and hiding the real fault. - Escalate for adding disks, the SSD tiering plan, or anything that needs a node reboot (each reboot degrades all ~18 volumes and triggers a rebuild storm). --- Source of truth: `runbooks/longhorn-capacity.md`. Index: `/runbooks/` Records: `platform/longhorn/helmrelease.yaml` (the settings above, the I/O throttle block, and the removal of the inert `orphanAutoDeletion`); `platform/cnpg/alerts.yaml` header (the WAL disk-full clock); `docs/BULK-NFS-STORAGE.md`.