0 Fork 0

Files

T

jakman85andClaude Opus 5 f663e7c801 fix(longhorn): the I/O caps a runbook promised were never declared — declare them

`runbooks/longhorn-degraded.md` told operators that, after the 2026-09-07 etcd
I/O-starvation incident, `concurrentVolumeBackupRestorePerNodeLimit` and
`snapshotHeavyTaskConcurrentLimit` had been tightened 5 -> 1. Measured live on
prod today: both read **5**, and neither name appeared anywhere in this
repository — so nothing declared them and nothing would have restored them.
Fourth instance of config that looked applied and was not.

Landed in `platform/longhorn/helmrelease.yaml` `defaultSettings` (the path Flux
reconciles; `values.yaml` is the reference mirror), NOT by `kubectl patch` —
a hand-applied Setting is exactly how this gap arose and Helm would silently
rewrite it on its next upgrade.

  concurrentVolumeBackupRestorePerNodeLimit      5 -> 1
  snapshotHeavyTaskConcurrentLimit              5 -> 1
  replicaReplenishmentWaitInterval              600     PINNED, unchanged
  backupConcurrentLimit / restoreConcurrentLimit  2 / 2 PINNED, unchanged
  concurrentAutomaticEngineUpgradePerNodeLimit  0       PINNED, unchanged
  nodeDrainPolicy  block-if-contains-last-replica       PINNED — it was in
      force only as Longhorn's OWN DEFAULT, declared in no manifest and in no
      rendered ConfigMap, so an upstream default change would have moved it
      silently while two runbooks reasoned about drain safety from it.

EVERY KEY PROVEN TO LAND, not assumed. `helm template longhorn/longhorn
--version 1.12.1` (the version this chart range resolves to, per the
HelmRelease's own lastAttemptedRevision) on this branch's spec.values,
substituted as clusters/prod substitutes them, emits all eight into the rendered
`longhorn-default-setting` ConfigMap in kebab-case. Diffed against the LIVE
ConfigMap the change is purely ADDITIVE: 8 keys added, none altered, none lost.
The chain to the Settings is verified read-only on prod — all 115
settings.longhorn.io carry `longhorn.io/configmap-resource-version: 7917`
(= that ConfigMap's resourceVersion) with `status.applied: true`, and every key
in the ConfigMap equals its Setting's value. The two targets sit at 5 with the
key absent, i.e. Longhorn defaults, as measured.

`orphanAutoDeletion: true` REMOVED, verified inert two independent ways against
the deployed 1.12.1: the render emits no orphan key at all (Longhorn 1.9 renamed
it `orphanResourceAutoDeletion`, a list, and the chart drops unknown
defaultSettings keys with no error), and live, `setting/orphan-auto-deletion` is
NotFound while `orphan-resource-auto-deletion` reads "". Deliberately NOT
replaced — orphan deletion is extra writes on the constrained partition and buys
only space. Orphan cleanup stays manual by decision; the two runbooks that cite
the key by name are corrected rather than left to go stale.

BOTH NEW CAPS ARE 1, AND 1 COSTS SOMETHING — stated, not waved at.
`concurrentVolumeBackupRestorePerNodeLimit: 1` costs nothing TODAY: Longhorn has
no backup target (the `backup-target` Setting does not exist; still the
WS-STORAGE Phase-4 TODO), and velero's nightly run is not a Longhorn backup —
`snapshotVolumes: false` + `defaultVolumesToFsBackup: true` is kopia
file-system backup through velero's node-agent, which this Setting does not
gate; CNPG base backups are barman-level and ungated too. Its only consumer is
an R-STATE-03 restore, where serialising ~24 volumes beats losing etcd
mid-restore. Revisit when a Longhorn backup target lands.
`snapshotHeavyTaskConcurrentLimit: 1` rather than 2 because purge work is
generated by rebuild cleanup (auto-cleanup-system-generated-snapshot true, NO
RecurringJobs, 24 volumes / 24 snapshots) and rebuild is already serialised to
1 per node — so 1 keeps pace with its own generator, where 2 buys backlog drain
at the price of a second heavy writer during the exact rebuild that starved
etcd. The cost: space reclaims serially after a rebuild storm — real here (a
37.7 GiB snapshot once pinned a volume at 97.3% allocated), but instrumented by
LonghornDiskAllocationHigh and recoverable by hand, where a wedged etcd is not.
Headroom on the record: I/O PSI full is 0.30/0.62/0.37 % now, and 3.79 under a
full 18-replica rebuild after node 1's drive replacement, against a 5% warning
threshold. These caps are a BOUND, not a fix — etcd and Longhorn still share one
ext4 filesystem and nothing arbitrates the jbd2 journal fdatasync waits on.

NOT PORTED, deliberately: the same unmerged branch's node-level I/O arbitration
(`roles/etcd_io_priority` — BFQ on the shared device, nr_requests 64 -> 32,
systemd IOWeight drop-ins). The 2011 C400 it was written for was replaced on
2026-09-07 and the same load class measured 11.36 -> 3.79 I/O PSI with the node
Ready throughout; the role was never run against any cluster, its author
recorded it as unproven with a node-3-first sequenced rollout, and its own
README concedes it does not separate the two tenants. Node-wide blast radius,
low ceiling, stale measurements (node 1 reinstalled, founder moved to node 2).
It stays an operator option if I/O PSI climbs back toward 5% — which, unlike in
September, we would now be told about. Nothing under infra/ansible/ is touched.

ALSO ALREADY DONE, so skipped: that branch's NodeIOPressureHigh/Critical and the
dashboard I/O row are on main already (efc6db6) with runbooks/node-io.md.
Verified live, not by reading the file: both rules loaded in
platform-health.nodes, health ok, state inactive, and
ALERTS{alertname=~"NodeIOPressure.*"} returns no samples.

Runbooks corrected in both directions: longhorn-degraded.md's prominent "DO NOT
RELY ON THAT: it was never applied" block becomes an accurate statement of what
is now declared, what to verify before trusting it, what it cannot do, and what
it costs; longhorn-capacity.md / node-disk.md stop citing a key that no longer
exists; node-io.md gains the one line an operator needs mid-incident.

No new ${...} substitution variables and not one dollar sign added anywhere in
the diff, so neither StrictPostBuildSubstitutions nor the L0 doubled-dollar guard
is affected.

Co-Authored-By: Claude Opus 5 <[email protected]>

2026-09-13 17:48:42 -05:00

7.7 KiB

Raw Blame History

node-disk — A node filesystem is filling up

Alert: NodeDiskFillingUp (warning, 15m) File: platform/monitoring/alerts.yaml, group platform-health.nodes Expression: node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"} / node_filesystem_size_bytes{...} < 0.10

What this means here

Less than 10% free on a real filesystem on a node, for 15 minutes. Read the mountpoint label first, because on these nodes it tells you which of two very different problems you have.

/data is the one that matters. Both the embedded etcd WAL (/data/k3s) and every Longhorn replica (/data/longhorn) live on that single partition — node_baseline_etcd_source: shared-data-partition, not a dedicated device, and the exit condition for splitting them is still open (infra/PROD-PROVISIONING.md §2). If /data fills:

etcd cannot write its WAL. A control-plane member with a full WAL partition does not degrade gracefully; it stops, and this is a 3-member cluster.
Longhorn stops being able to hold replicas and volumes go degraded, which triggers rebuilds, which need more space.

So /data at 10% free is not a warning, it is the last comfortable moment to act.

/ filling up is usually logs, container images or a stuck journal. Lower stakes, but there is history: in the dev era the host root filled to ~99% because VM disks sat under a snapper-snapshotted subvolume, and the hypervisor paused the k3s VMs on I/O error. Recovery was deleting 11 snapper timeline snapshots plus virsh destroy on the paused VMs (which were holding rsync-unlinked inodes): 99% → 30%.

Longhorn's own guard rails are already tight and are why this alert usually fires before anything breaks: storageReservedPercentageForDefaultDisk: 20, storageMinimalAvailablePercentage: 25, storageOverProvisioningPercentage: 200.

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two dev.sh invocations at once; they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git (git push origin main is the deploy). Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

Which filesystem, on which node:

ssh k3s-prod-1 'df -h -x tmpfs -x overlay; echo; du -xh --max-depth=1 /data 2>/dev/null | sort -h | tail -10; echo; du -xh --max-depth=1 / 2>/dev/null | sort -h | tail -10'

What Longhorn thinks it has:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o custom-columns=NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,SCHED:.spec.allowScheduling; echo; \
   kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,SIZE:.spec.size,ACTUAL:.status.actualSize,ROBUST:.status.robustness'

Journal and image bloat on /:

ssh k3s-prod-1 'sudo journalctl --disk-usage; sudo crictl images | wc -l; sudo du -sh /var/lib/rancher/k3s/agent/containerd 2>/dev/null'

Likely causes, most likely first

1. Longhorn replicas grew, or replicas landed here after a rebuild

Hard node anti-affinity (replicaSoftAntiAffinity: false) means one replica per node per volume, but replicaAutoBalance: best-effort moves them. Prod sits around 269 GiB scheduled against an 829 GiB ceiling, so this is rarely tight — but a rebuild storm plus snapshots can spike it. Remediation: check for orphaned replicas and stale snapshots in the Longhorn UI. Orphan cleanup is manual: orphanAutoDeletion: true sat in platform/longhorn/helmrelease.yaml and had been silently inert since Longhorn 1.12 — renamed orphanResourceAutoDeletion (a list), and the chart drops unknown defaultSettings keys with no error. Removed 2026-09-13 and deliberately not replaced. Snapshot purge also runs one at a time per node now (snapshotHeavyTaskConcurrentLimit: 1, the etcd WAL protection), so reclaim after a rebuild is serial by design.

2. A CNPG database is not recycling WAL

The single most dangerous cause, because it is unbounded. With archive_mode=on and a failing archive_command, Postgres refuses to recycle unarchived WAL segments and pg_wal grows forever — on prod's keycloak-db that was measured at ~6.8 MiB/h, 17 segments / ~268 MiB, about 29 days to fill a 5 GiB PVC. pg_wal shares the data PVC (no walStorage: is configured on any Cluster). Check /runbooks/cnpg-archiving before you free any space, and never delete WAL by hand — that destroys the database.

3. Journal / container images on `/`

journalctl --vacuum-size=200M and pruning unused images are safe. Do not prune images for workloads that may need to restart while the registry is unreachable.

4. A Velero or CDI staging write

Velero's kopia uploads and CDI's scratch space both stage locally. longhorn-scratch (single-replica) exists precisely because the 3-replica default failed the UniFi 60Gi import on insufficient storage.

5. The disk is smaller than the platform now needs

docs/plans/longhorn-ssd-tiering.md holds the planned hot tier (~51 GB: Prometheus, CNPG, Loki, Gitea). Working set measured ~79 GB against ~323 GB free on SSD, so the "won't fit" blocker was a measurement artifact. This is operator planning work, not an incident fix.

When to escalate or stop

If the mountpoint is /data and free space is under 5%, treat it as an imminent control-plane outage and get the operator. etcd and Longhorn are both on it.
Never delete anything under /data/longhorn or a Postgres pg_wal directory by hand. Both are irreversible. Free space elsewhere, or fix the cause.
Do not kubectl drain the node to move load off it — this cluster does not drain.
If the cause is a CNPG archive failure, this runbook is the wrong one: go to /runbooks/cnpg-archiving, which is the actual fix.

Source of truth: runbooks/node-disk.md. Index: /runbooks/ Records: DECISIONS-LOG.md "Disk/drive incident + k3s relocated to NVMe (2026-08-22)"; platform/cnpg/alerts.yaml header (the WAL disk-full clock).

7.7 KiB Raw Blame History