home-network-k3s-impl/runbooks/longhorn-capacity.md

longhorn-capacity — A Longhorn disk is running out of room for replicas

Alerts: LonghornDiskAllocationHigh (warning, 30m) and LonghornDiskUnschedulable (warning, 15m) File: platform/monitoring/alerts.yaml, group platform-health.longhorn Expressions: (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70 longhorn_disk_status{condition="schedulable"} == 0

What this means here

This is a POOL question, not a PVC question. One of the three nodes' Longhorn data disks is filling with replica files, or Longhorn has already stopped placing replicas on it. Neither alert says anything about how full any single workload's filesystem is — for that, read /runbooks/pvc-capacity (PersistentVolumeFillingUp, PersistentVolumeAlmostFull) or /runbooks/monitoring-capacity (MonitoringVolumeFillingUp). Those are keyed on kubelet_volume_stats_*, which is the filesystem's own accounting.

longhorn_disk_usage_bytes counts bytes ALLOCATED: every replica file plus every snapshot still pinned to it. Longhorn does not return freed blocks to a replica file without discard/TRIM, and a system snapshot taken for a replica rebuild keeps its historical blocks alive, so this number only goes up until something is deleted. That is the correct question here — allocated bytes are what consumes the node's ~414 GiB data partition — and the wrong question per volume, which is why LonghornVolumeCapacityHigh was deleted on 2026-09-13 (it read 0.973 on a Prometheus TSDB volume whose filesystem was 20.8% used, by design, permanently). See the comment block in platform/monitoring/alerts.yaml for the measurements.

Why 0.70 and not 0.85. storageMinimalAvailablePercentage: 25 means Longhorn itself refuses to schedule a new replica on a disk with under 25% available — i.e. at 75% used the disk is already out of the scheduling pool. An 85% threshold could only ever fire after that had happened. 0.70 is the last point at which there is still lead time. The companion settings are storageReservedPercentageForDefaultDisk: 20, storageOverProvisioningPercentage: 200, defaultDataLocality: disabled, hard node anti-affinity.

Measured on prod 2026-09-13: usage 0.2254 / 0.2306 / 0.2369 on prod-1/2/3, with 268.9 GiB of nominal replica size scheduled against (414.5 GiB − 20% reserved) × 200%. So either alert firing means something changed materially, not that the platform slowly grew.

The severe consequence is that a degraded volume cannot finish rebuilding — hard anti-affinity needs three distinct nodes with room, so one full or unschedulable disk pins a volume at 2 replicas indefinitely, and R-STATE-02's survive-a-node-loss promise is suspended for as long as that lasts. Cross-read /runbooks/longhorn-degraded.

Verify the alert can fire (measured 2026-09-13)

These alerts were INERT until 2026-09-13, when metrics.serviceMonitor.enabled was set in platform/longhorn/helmrelease.yaml; the longhorn_* family was simply absent from the TSDB before that. Both are now live — verified on prod the same day:

infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \
  prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
  /bin/promtool query instant http://localhost:9090 \
  'count(longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) + count(longhorn_disk_status{condition="schedulable"})'

returned 6 (3 disks, both selectors resolving), while both alert expressions returned empty. If that count is ever empty again, the scrape has broken and this pair is silently back to inert — check up{job="longhorn-backend"} and platform/longhorn/networkpolicy-metrics.yaml.

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two dev.sh invocations at once; they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git. main is a protected branch — a direct push is rejected. A change lands by pushing a branch, opening a PR, and merging it once the gate is green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

Per-disk allocation and Longhorn's own scheduling verdict — the two things the alerts read:

infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \
  prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
  /bin/promtool query instant http://localhost:9090 \
  'longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes or longhorn_disk_status'

Node-level allocation, from the CRs:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o yaml | grep -E "name:|storageAvailable|storageMaximum|storageScheduled|allowScheduling"'

Volume nominal size versus ALLOCATED size. actualSize is head + pinned snapshots, NOT filesystem usage — a volume can read near 100% here with a nearly empty filesystem (see the deleted-alert note above), so do not read this column as "full":

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
     -o custom-columns=NAME:.metadata.name,SPEC:.spec.size,ACTUAL:.status.actualSize,ROBUST:.status.robustness'

Which PVC each one is, so you know who is filling it:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get pv -o custom-columns=PV:.metadata.name,CLAIM:.spec.claimRef.namespace/.spec.claimRef.name,SIZE:.spec.capacity.storage,SC:.spec.storageClassName'

Which snapshots are holding the allocation (this is the measured top cause on prod — one user_created=false system snapshot on the Prometheus TSDB volume held 37.7 GiB on 2026-09-13):

infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \
  prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
  /bin/promtool query instant http://localhost:9090 \
  'topk(10, longhorn_snapshot_actual_size_bytes)'

The filesystem view — the separate question, answered by /runbooks/pvc-capacity:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   curl -s -G --data-urlencode "query=topk(10, 1 - kubelet_volume_stats_available_bytes{job=\"kubelet\"} / kubelet_volume_stats_capacity_bytes{job=\"kubelet\"})" localhost:19090/api/v1/query'

On the node itself:

ssh k3s-prod-1 'df -h /data; du -xh --max-depth=1 /data/longhorn/replicas 2>/dev/null | sort -h | tail'

Likely causes, most likely first

0. Pinned snapshots, not live data (check this FIRST for an allocation alert)

An allocation alert can fire with every filesystem in the cluster nearly empty. Run the longhorn_snapshot_actual_size_bytes query above before anything else. A replica rebuild — which is what a node reboot or a disk replacement forces, and prod-1's drive was replaced on 2026-09-12 — leaves a user_created=false system snapshot behind, and it keeps its blocks until it is purged. Prune from the Longhorn UI (Volume → Snapshots), then re-read the query; nothing else is needed if this was the cause.

1. A CNPG database is not recycling WAL (check this FIRST for a PVC-fill alert)

The only cause here that grows without bound. archive_mode=on plus a failing archive_command means Postgres will not recycle unarchived WAL; measured on prod keycloak-db at ~6.8 MiB/h with pg_wal sharing the data PVC (no walStorage: is configured on any Cluster) — about 29 days from 93% free to disk-full, ending in a postmaster PANIC. Go to /runbooks/cnpg-archiving. Never delete WAL by hand.

2. Snapshots and orphaned replicas accumulating

Longhorn snapshots are not free, and cleanup here is manual. orphanAutoDeletion: true used to sit in platform/longhorn/helmrelease.yaml and had been silently inert since Longhorn 1.12 — it was renamed orphanResourceAutoDeletion (now a list) and the chart drops unknown defaultSettings keys with no error. It was removed on 2026-09-13 and deliberately not replaced, so orphan cleanup is manual by decision, not by accident: prune orphans and stale snapshots from the Longhorn UI.

Note also that snapshot purge and clone now run one at a time per node (snapshotHeavyTaskConcurrentLimit: 1, declared 2026-09-13 to keep Longhorn from starving etcd's WAL fsync). Reclaim after a rebuild storm is therefore serial and slower by design. If allocation is climbing and purge is the bottleneck, the operator may raise the Setting for the duration and watch NodeIOPressureHigh — see runbooks/longhorn-degraded.md for the tradeoff as recorded.

3. A workload genuinely grew

Plex/Sonarr/Radarr media does not live on Longhorn — it is static NFS from 192.168.2.102:/mnt/Storage/plex, via the bulk tier. If a media volume is on longhorn, that is a misconfiguration worth fixing, not a capacity problem: the bulk StorageClass is a stub (provisioner: nfs.csi.k8s.io, server:/share: REPLACE-ME) and provisions nothing, so real bulk storage is static NFS PVs. Otherwise expand the PVC — all Longhorn classes have allowVolumeExpansion: true; change the size in git and push.

4. Replicas piled onto one node

replicaAutoBalance: best-effort should prevent this, and hard anti-affinity bounds it to one replica per volume per node. If one node is much fuller than the others, check whether a rebuild placed replicas there while another node was down.

5. The cluster has outgrown its disks

Prod's ceiling is ~829 GiB across three nodes. docs/plans/longhorn-ssd-tiering.md is the planned answer (a ~51 GB hot tier: Prometheus, CNPG, Loki, Gitea). Operator planning work.

When to escalate or stop

• A disk over 70% while any volume is degraded is urgent, because the rebuild cannot complete — and at 75% Longhorn takes the disk out of the scheduling pool entirely (LonghornDiskUnschedulable). Free space or restore the other node before anything else.
• Never delete files under /data/longhorn or inside a Postgres pg_wal. Both are irreversible.
• Do not expand a PVC as a reflex if the cause is unrecycled WAL — you are buying days and hiding the real fault.
• Escalate for adding disks, the SSD tiering plan, or anything that needs a node reboot (each reboot degrades all ~18 volumes and triggers a rebuild storm).

---

Source of truth: runbooks/longhorn-capacity.md. Index: /runbooks/ Records: platform/longhorn/helmrelease.yaml (the settings above, the I/O throttle block, and the removal of the inert orphanAutoDeletion); platform/cnpg/alerts.yaml header (the WAL disk-full clock); docs/BULK-NFS-STORAGE.md.