home-network-k3s-impl/runbooks/node-disk.md

node-disk — A node filesystem is filling up

Alert: NodeDiskFillingUp (warning, 15m) File: platform/monitoring/alerts.yaml, group platform-health.nodes Expression: node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"} / node_filesystem_size_bytes{...} < 0.10

What this means here

Less than 10% free on a real filesystem on a node, for 15 minutes. Read the mountpoint label first, because on these nodes it tells you which of two very different problems you have.

/data is the one that matters. Both the embedded etcd WAL (/data/k3s) and every Longhorn replica (/data/longhorn) live on that single partition — node_baseline_etcd_source: shared-data-partition, not a dedicated device, and the exit condition for splitting them is still open (infra/PROD-PROVISIONING.md §2). If /data fills:

• etcd cannot write its WAL. A control-plane member with a full WAL partition does not degrade gracefully; it stops, and this is a 3-member cluster.
• Longhorn stops being able to hold replicas and volumes go degraded, which triggers rebuilds, which need more space.

So /data at 10% free is not a warning, it is the last comfortable moment to act.

/ filling up is usually logs, container images or a stuck journal. Lower stakes, but there is history: in the dev era the host root filled to ~99% because VM disks sat under a snapper-snapshotted subvolume, and the hypervisor paused the k3s VMs on I/O error. Recovery was deleting 11 snapper timeline snapshots plus virsh destroy on the paused VMs (which were holding rsync-unlinked inodes): 99% → 30%.

Longhorn's own guard rails are already tight and are why this alert usually fires before anything breaks: storageReservedPercentageForDefaultDisk: 20, storageMinimalAvailablePercentage: 25, storageOverProvisioningPercentage: 200.

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two dev.sh invocations at once; they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git (git push origin main is the deploy). Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

Which filesystem, on which node:

ssh k3s-prod-1 'df -h -x tmpfs -x overlay; echo; du -xh --max-depth=1 /data 2>/dev/null | sort -h | tail -10; echo; du -xh --max-depth=1 / 2>/dev/null | sort -h | tail -10'

What Longhorn thinks it has:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o custom-columns=NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,SCHED:.spec.allowScheduling; echo; \
   kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,SIZE:.spec.size,ACTUAL:.status.actualSize,ROBUST:.status.robustness'

Journal and image bloat on /:

ssh k3s-prod-1 'sudo journalctl --disk-usage; sudo crictl images | wc -l; sudo du -sh /var/lib/rancher/k3s/agent/containerd 2>/dev/null'

Likely causes, most likely first

1. Longhorn replicas grew, or replicas landed here after a rebuild

Hard node anti-affinity (replicaSoftAntiAffinity: false) means one replica per node per volume, but replicaAutoBalance: best-effort moves them. Prod sits around 269 GiB scheduled against an 829 GiB ceiling, so this is rarely tight — but a rebuild storm plus snapshots can spike it. Remediation: check for orphaned replicas and stale snapshots in the Longhorn UI. Note orphanAutoDeletion: true in platform/longhorn/helmrelease.yaml has been silently inert since Longhorn 1.12 — it was renamed orphanResourceAutoDeletion (a list) and the chart drops unknown defaultSettings keys with no error. So orphan cleanup is manual. Verify the key's current state in the file before you trust it.

2. A CNPG database is not recycling WAL

The single most dangerous cause, because it is unbounded. With archive_mode=on and a failing archive_command, Postgres refuses to recycle unarchived WAL segments and pg_wal grows forever — on prod's keycloak-db that was measured at ~6.8 MiB/h, 17 segments / ~268 MiB, about 29 days to fill a 5 GiB PVC. pg_wal shares the data PVC (no walStorage: is configured on any Cluster). Check /runbooks/cnpg-archiving before you free any space, and never delete WAL by hand — that destroys the database.

3. Journal / container images on /

journalctl --vacuum-size=200M and pruning unused images are safe. Do not prune images for workloads that may need to restart while the registry is unreachable.

4. A Velero or CDI staging write

Velero's kopia uploads and CDI's scratch space both stage locally. longhorn-scratch (single-replica) exists precisely because the 3-replica default failed the UniFi 60Gi import on insufficient storage.

5. The disk is smaller than the platform now needs

docs/plans/longhorn-ssd-tiering.md holds the planned hot tier (~51 GB: Prometheus, CNPG, Loki, Gitea). Working set measured ~79 GB against ~323 GB free on SSD, so the "won't fit" blocker was a measurement artifact. This is operator planning work, not an incident fix.

When to escalate or stop

• If the mountpoint is /data and free space is under 5%, treat it as an imminent control-plane outage and get the operator. etcd and Longhorn are both on it.
• Never delete anything under /data/longhorn or a Postgres pg_wal directory by hand. Both are irreversible. Free space elsewhere, or fix the cause.
• Do not kubectl drain the node to move load off it — this cluster does not drain.
• If the cause is a CNPG archive failure, this runbook is the wrong one: go to /runbooks/cnpg-archiving, which is the actual fix.

---

Source of truth: runbooks/node-disk.md. Index: /runbooks/ Records: DECISIONS-LOG.md "Disk/drive incident + k3s relocated to NVMe (2026-08-22)"; platform/cnpg/alerts.yaml header (the WAL disk-full clock).