home-network-k3s-impl/runbooks/node-disk.md

node-disk — A node filesystem is filling up

Alert: NodeDiskFillingUp (warning, 15m) File: platform/monitoring/alerts.yaml, group platform-health.nodes Expression: node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"} / node_filesystem_size_bytes{...} < 0.10

What this means here

Less than 10% free on a real filesystem on a node, for 15 minutes. Read the mountpoint label first, because on these nodes it tells you which of two very different problems you have.

/data is the one that matters. Both the embedded etcd WAL (/data/k3s) and every Longhorn replica (/data/longhorn) live on that single partition — node_baseline_etcd_source: shared-data-partition, not a dedicated device, and the exit condition for splitting them is still open (infra/PROD-PROVISIONING.md §2). If /data fills:

• etcd cannot write its WAL. A control-plane member with a full WAL partition does not degrade gracefully; it stops, and this is a 3-member cluster.
• Longhorn stops being able to hold replicas and volumes go degraded, which triggers rebuilds, which need more space.

So /data at 10% free is not a warning, it is the last comfortable moment to act.

/ filling up is usually logs, container images or a stuck journal. Lower stakes, but there is history: in the dev era the host root filled to ~99% because VM disks sat under a snapper-snapshotted subvolume, and the hypervisor paused the k3s VMs on I/O error. Recovery was deleting 11 snapper timeline snapshots plus virsh destroy on the paused VMs (which were holding rsync-unlinked inodes): 99% → 30%.

Longhorn's own guard rails are already tight and are why this alert usually fires before anything breaks: storageReservedPercentageForDefaultDisk: 20, storageMinimalAvailablePercentage: 25, storageOverProvisioningPercentage: 200.

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two dev.sh invocations at once; they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git (git push origin main is the deploy). Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

Which filesystem, on which node:

ssh k3s-prod-1 'df -h -x tmpfs -x overlay; echo; du -xh --max-depth=1 /data 2>/dev/null | sort -h | tail -10; echo; du -xh --max-depth=1 / 2>/dev/null | sort -h | tail -10'

What Longhorn thinks it has:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o custom-columns=NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,SCHED:.spec.allowScheduling; echo; \
   kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,SIZE:.spec.size,ACTUAL:.status.actualSize,ROBUST:.status.robustness'

Journal and image bloat on /:

ssh k3s-prod-1 'sudo journalctl --disk-usage; sudo crictl images | wc -l; sudo du -sh /var/lib/rancher/k3s/agent/containerd 2>/dev/null'

Likely causes, most likely first

1. Longhorn replicas grew, or replicas landed here after a rebuild

Hard node anti-affinity (replicaSoftAntiAffinity: false) means one replica per node per volume, but replicaAutoBalance: best-effort moves them. Prod sits around 269 GiB scheduled against an 829 GiB ceiling, so this is rarely tight — but a rebuild storm plus snapshots can spike it. Remediation: check for orphaned replicas and stale snapshots in the Longhorn UI. Orphan cleanup is manual: orphanAutoDeletion: true sat in platform/longhorn/helmrelease.yaml and had been silently inert since Longhorn 1.12 — renamed orphanResourceAutoDeletion (a list), and the chart drops unknown defaultSettings keys with no error. Removed 2026-09-13 and deliberately not replaced. Snapshot purge also runs one at a time per node now (snapshotHeavyTaskConcurrentLimit: 1, the etcd WAL protection), so reclaim after a rebuild is serial by design.

2. A CNPG database is not recycling WAL

The single most dangerous cause, because it is unbounded. With archive_mode=on and a failing archive_command, Postgres refuses to recycle unarchived WAL segments and pg_wal grows forever — on prod's keycloak-db that was measured at ~6.8 MiB/h, 17 segments / ~268 MiB, about 29 days to fill a 5 GiB PVC. pg_wal shares the data PVC (no walStorage: is configured on any Cluster). Check /runbooks/cnpg-archiving before you free any space, and never delete WAL by hand — that destroys the database.

3. Journal / container images on /

journalctl --vacuum-size=200M and pruning unused images are safe. Do not prune images for workloads that may need to restart while the registry is unreachable.

4. A Velero or CDI staging write

Velero's kopia uploads and CDI's scratch space both stage locally. longhorn-scratch (single-replica) exists precisely because the 3-replica default failed the UniFi 60Gi import on insufficient storage.

5. The disk is smaller than the platform now needs

docs/plans/longhorn-ssd-tiering.md holds the planned hot tier (~51 GB: Prometheus, CNPG, Loki, Gitea). Working set measured ~79 GB against ~323 GB free on SSD, so the "won't fit" blocker was a measurement artifact. This is operator planning work, not an incident fix.

When to escalate or stop

• If the mountpoint is /data and free space is under 5%, treat it as an imminent control-plane outage and get the operator. etcd and Longhorn are both on it.
• Never delete anything under /data/longhorn or a Postgres pg_wal directory by hand. Both are irreversible. Free space elsewhere, or fix the cause.
• Do not kubectl drain the node to move load off it — this cluster does not drain.
• If the cause is a CNPG archive failure, this runbook is the wrong one: go to /runbooks/cnpg-archiving, which is the actual fix.

---

Source of truth: runbooks/node-disk.md. Index: /runbooks/ Records: DECISIONS-LOG.md "Disk/drive incident + k3s relocated to NVMe (2026-08-22)"; platform/cnpg/alerts.yaml header (the WAL disk-full clock).