home-network-k3s-impl/runbooks/node-disk.md

# node-disk — A node filesystem is filling up

**Alert:** `NodeDiskFillingUp` (warning, 15m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes`
**Expression:** `node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"} / node_filesystem_size_bytes{...} < 0.10`

## What this means here

Less than 10% free on a real filesystem on a node, for 15 minutes. Read the `mountpoint` label
first, because on these nodes it tells you which of two very different problems you have.

**`/data` is the one that matters.** Both the embedded **etcd WAL** (`/data/k3s`) and **every
Longhorn replica** (`/data/longhorn`) live on that single partition —
`node_baseline_etcd_source: shared-data-partition`, not a dedicated device, and the exit condition
for splitting them is still open (`infra/PROD-PROVISIONING.md` §2). If `/data` fills:

- etcd cannot write its WAL. A control-plane member with a full WAL partition does not degrade
  gracefully; it stops, and this is a 3-member cluster.
- Longhorn stops being able to hold replicas and volumes go degraded, which triggers rebuilds, which
  need *more* space.

So `/data` at 10% free is not a warning, it is the last comfortable moment to act.

**`/` filling up** is usually logs, container images or a stuck journal. Lower stakes, but there is
history: in the dev era the host root filled to ~99% because VM disks sat under a snapper-snapshotted
subvolume, and the hypervisor **paused the k3s VMs on I/O error**. Recovery was deleting 11 snapper
timeline snapshots plus `virsh destroy` on the paused VMs (which were holding rsync-unlinked inodes):
99% → 30%.

Longhorn's own guard rails are already tight and are why this alert usually fires before anything
breaks: `storageReservedPercentageForDefaultDisk: 20`,
`storageMinimalAvailablePercentage: 25`, `storageOverProvisioningPercentage: 200`.
## Running the commands in this runbook

`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.

## First commands

Which filesystem, on which node:

```
ssh k3s-prod-1 'df -h -x tmpfs -x overlay; echo; du -xh --max-depth=1 /data 2>/dev/null | sort -h | tail -10; echo; du -xh --max-depth=1 / 2>/dev/null | sort -h | tail -10'
```

What Longhorn thinks it has:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o custom-columns=NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,SCHED:.spec.allowScheduling; echo; \
   kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,SIZE:.spec.size,ACTUAL:.status.actualSize,ROBUST:.status.robustness'
```

Journal and image bloat on `/`:

```
ssh k3s-prod-1 'sudo journalctl --disk-usage; sudo crictl images | wc -l; sudo du -sh /var/lib/rancher/k3s/agent/containerd 2>/dev/null'
```

## Likely causes, most likely first

### 1. Longhorn replicas grew, or replicas landed here after a rebuild
Hard node anti-affinity (`replicaSoftAntiAffinity: false`) means one replica per node per volume, but
`replicaAutoBalance: best-effort` moves them. Prod sits around 269 GiB scheduled against an 829 GiB
ceiling, so this is rarely tight — but a rebuild storm plus snapshots can spike it.
Remediation: check for orphaned replicas and stale snapshots in the Longhorn UI. Note
`orphanAutoDeletion: true` in `platform/longhorn/helmrelease.yaml` **has been silently inert since
Longhorn 1.12** — it was renamed `orphanResourceAutoDeletion` (a list) and the chart drops unknown
`defaultSettings` keys with no error. So orphan cleanup is manual. Verify the key's current state in
the file before you trust it.

### 2. A CNPG database is not recycling WAL
The single most dangerous cause, because it is unbounded. With `archive_mode=on` and a failing
`archive_command`, Postgres **refuses to recycle** unarchived WAL segments and `pg_wal` grows forever
— on prod's `keycloak-db` that was measured at ~6.8 MiB/h, 17 segments / ~268 MiB, about 29 days to
fill a 5 GiB PVC. `pg_wal` shares the data PVC (no `walStorage:` is configured on any Cluster).
Check `/runbooks/cnpg-archiving` **before** you free any space, and **never delete WAL by hand** —
that destroys the database.

### 3. Journal / container images on `/`
`journalctl --vacuum-size=200M` and pruning unused images are safe. Do not prune images for
workloads that may need to restart while the registry is unreachable.

### 4. A Velero or CDI staging write
Velero's kopia uploads and CDI's scratch space both stage locally. `longhorn-scratch`
(single-replica) exists precisely because the 3-replica default failed the UniFi 60Gi import on
insufficient storage.

### 5. The disk is smaller than the platform now needs
`docs/plans/longhorn-ssd-tiering.md` holds the planned hot tier (~51 GB: Prometheus, CNPG, Loki,
Gitea). Working set measured ~79 GB against ~323 GB free on SSD, so the "won't fit" blocker was a
measurement artifact. This is operator planning work, not an incident fix.

## When to escalate or stop

- **If the mountpoint is `/data` and free space is under 5%, treat it as an imminent control-plane
  outage** and get the operator. etcd and Longhorn are both on it.
- **Never delete anything under `/data/longhorn` or a Postgres `pg_wal` directory by hand.** Both are
  irreversible. Free space elsewhere, or fix the cause.
- **Do not `kubectl drain`** the node to move load off it — this cluster does not drain.
- If the cause is a CNPG archive failure, this runbook is the wrong one: go to
  `/runbooks/cnpg-archiving`, which is the actual fix.

---
Source of truth: `runbooks/node-disk.md`. Index: `/runbooks/`
Records: `DECISIONS-LOG.md` "Disk/drive incident + k3s relocated to NVMe (2026-08-22)";
`platform/cnpg/alerts.yaml` header (the WAL disk-full clock).