home-network-k3s-impl/runbooks/longhorn-degraded.md

longhorn-degraded — A Longhorn volume has been degraded for 10 minutes

Alert: LonghornVolumeDegraded (warning, 10m) File: platform/monitoring/alerts.yaml, group platform-health.longhorn Expression: longhorn_volume_robustness{state="degraded"} == 1 (one-hot over state, NOT a 0-3 enum)

What this means here

A volume has fewer healthy replicas than it should. It is still serving I/O — that is the difference between degraded and faulted (/runbooks/longhorn-faulted). What you have lost is redundancy, and on a 3-node cluster with hard node anti-affinity (replicaSoftAntiAffinity: false, numberOfReplicas: 3) one degraded volume means one more node loss away from data loss.

Degraded is the NORMAL state after any node reboot on this cluster, and it is expensive. Recorded plainly in the stabilization notes: "Longhorn on the HDD: every node reboot degrades all 18 volumes → full rebuild (slow; 30%+ IO-delay). Roll reboots one node at a time; wait robustness=healthy before the next." The rebuild is the thing that hurts, not the degradation: rebuild writes land on the same partition as the embedded etcd WAL (/data/k3s and /data/longhorn share one device), and that is the mechanism behind the 2026-09-07 k3s-prod-1 NotReady outage. So the question to answer is never just "why is it degraded" but "is the rebuild about to take a node out". Cross-read /runbooks/node-io.

Verify the alert can fire (measured 2026-09-13, and it was inert TWICE)

This rule was inert for two independent reasons, both fixed on 2026-09-13, and either one alone was enough to make a faulted or degraded volume alert nobody:

1. No scrape. count(longhorn_volume_robustness) returned 0 series — the longhorn_* family was absent from the TSDB because the chart's metrics.serviceMonitor.enabled defaulted false. Fixed in platform/longhorn/helmrelease.yaml.
2. Wrong metric shape. With the series present the rule STILL could not fire. It read longhorn_volume_robustness == 2 on the assumption of a 0–3 enum; on longhorn 1.12.1 the metric is one-hot over a state label with values 0/1 only. Measured on prod the same day: count(longhorn_volume_robustness) = 96 (24 volumes × 4 states), count by (state) (longhorn_volume_robustness == 1) = {state="healthy"} 24, and count(longhorn_volume_robustness == 2) = empty. Fixed by keying on the label.

Both directions are now pinned offline by platform/monitoring/tests/longhorn.test.yaml, which keeps a regression case asserting the old == 2 form matches nothing. Check the live series before trusting this alert's silence:

infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \
  prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
  /bin/promtool query instant http://localhost:9090 \
  'count by (state) (longhorn_volume_robustness)'

Four states, 24 each, is healthy wiring. Empty means the scrape has broken and this alert is inert again — do not infer "no alert, so volumes are fine".

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two dev.sh invocations at once; they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git. main is a protected branch — a direct push is rejected. A change lands by pushing a branch, opening a PR, and merging it once the gate is green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

The truth about robustness, straight from the CRs (this works whether or not the metric does):

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
     -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,STATE:.status.state,NODE:.status.currentNodeID,SIZE:.spec.size'

Which replicas are missing, and where:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system get replicas.longhorn.io \
     -o custom-columns=NAME:.metadata.name,VOL:.spec.volumeName,NODE:.spec.nodeID,RUNNING:.status.currentState | sort -k2'

Which PVC/workload is affected:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get pv -o custom-columns=PV:.metadata.name,CLAIM:.spec.claimRef.namespace/.spec.claimRef.name,SC:.spec.storageClassName'

Are the nodes and their disks schedulable, and is a rebuild in flight?

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o wide; echo; \
   kubectl --context k3s-prod -n longhorn-system get settings.longhorn.io concurrent-replica-rebuild-per-node-limit -o custom-columns=NAME:.metadata.name,VALUE:.value'

And the pressure that rebuild is causing — the reason this alert matters:

ssh k3s-prod-1 'cat /proc/pressure/io'

Likely causes, most likely first

1. A node rebooted or was briefly NotReady (overwhelmingly the most common)

Expected, and it self-heals by rebuilding. Your job is to keep the rebuild from taking a node down. Remediation (operator, mutating) — freeze rebuilds while replicas will reattach (an in-place bounce):

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system patch settings.longhorn.io \
     concurrent-replica-rebuild-per-node-limit --type=merge -p "{\"value\":\"0\"}"'

Thaw to "1" — never higher — once every volume reads healthy. Use 1 rather than 0 when the replicas are genuinely gone (an OS re-provision), so replenishment can actually proceed. concurrentReplicaRebuildPerNodeLimit: 1 was already in force during the 2026-09-07 outage and held; do not "fix" this by setting a knob that is already set. replicaRebuildingBandwidthLimit is v2/SPDK-only and does nothing here, and there is no rebuild-concurrent-sync-limit Setting on 1.12.1 at all. What the incident actually called for was tightening every other Longhorn I/O source — and as of 2026-09-13 that is declared in platform/longhorn/helmrelease.yaml (defaultSettings), not just intended:

setting	declared	was, before
concurrent-replica-rebuild-per-node-limit	1	1 (already declared)
concurrent-volume-backup-restore-per-node-limit	1	5 (Longhorn default, undeclared)
snapshot-heavy-task-concurrent-limit	1	5 (Longhorn default, undeclared)
replica-replenishment-wait-interval	600	600 (pinned, not changed)
backup-concurrent-limit / restore-concurrent-limit	2 / 2	2 / 2 (pinned)
concurrent-automatic-engine-upgrade-per-node-limit	0	0 (pinned)
node-drain-policy	block-if-contains-last-replica	same, but as Longhorn's default only

WHAT CHANGED, AND WHAT TO CHECK BEFORE TRUSTING IT. Until 2026-09-13 this page asserted a cap that did not exist: snapshot-heavy-task-concurrent-limit and concurrent-volume-backup-restore-per-node-limit both measured 5 on prod, neither name appeared anywhere in the repository, and the work sat on an unmerged branch. They are now declared in the HelmRelease and reconciled by Flux, which is what makes them survive a rebuild — a kubectl patch is how the gap arose in the first place.

Verify, do not assume (the whole point of this note): kubectl -n longhorn-system get setting <name> -o jsonpath='{.value}'. If a value reads 5, the HelmRelease has not reconciled — check kubectl -n longhorn-system get helmrelease longhorn and that longhorn-default-setting carries the key (kubectl -n longhorn-system get cm longhorn-default-setting -o jsonpath='{.data}'). The chart drops unknown defaultSettings keys with no error, so "the key is in git" is not the same as "the key is in effect" — orphanAutoDeletion was inert here for two minor versions for exactly that reason, and has now been removed.

These caps are a bound, not a fix. They limit how much competing I/O Longhorn is allowed to generate. etcd's WAL and every replica still share one partition on every prod node, and no scheduler or cgroup control arbitrates the ext4 journal that fdatasync waits on. The measurable improvement came from hardware: node 1's 2011 SATA SSD was replaced, and under the same load class (an 18-replica rebuild with node 1 as the write target) I/O PSI full fell 11.36 -> 3.79 and the node stayed Ready. 3.79 against the 5% NodeIOPressureHigh threshold is the headroom you have.

Cost of the two caps, so nobody is surprised during an incident: snapshot purge and clone run one at a time per node, so space is reclaimed serially after a rebuild storm (watch LonghornDiskAllocationHigh); and a Longhorn volume restore runs one at a time per node. The latter costs nothing today — no Longhorn backup target is configured, and velero's nightly run is kopia file-system backup (defaultVolumesToFsBackup: true, snapshotVolumes: false), which this setting does not gate. During a real DR restore an operator may raise concurrent-volume-backup-restore-per-node-limit for the duration and watch NodeIOPressureHigh — that is a deliberate, temporary patch, like the rebuild freeze above.

Not ported, deliberately: the same unmerged branch also carried node-level I/O arbitration (BFQ on the shared device, nr_requests 64 -> 32, systemd IOWeight drop-ins; infra/ansible/roles/etcd_io_priority). It was never run against any cluster, its own author recorded it as unproven and node-3-first, and the drive it was written for has since been replaced. It remains an operator option if I/O PSI climbs back toward 5%, not a merged default.

2. A node is still down

Then the rebuild cannot complete and this will not clear. Go to /runbooks/node-down first; there is nothing to fix in Longhorn.

3. A disk was replaced and Longhorn refuses it

Signature: DiskFilesystemChanged — "record diskUUID doesn't match the one on the disk". Remediation, in exactly this order (the validating webhook enforces it): disable the disk → remove the disk entry → re-add it. Seen during the 2026-09-07 drive swap.

4. No node can hold the replica (anti-affinity or space)

Hard anti-affinity means a 3-replica volume needs three distinct schedulable nodes. With one node cordoned or its disk over storageMinimalAvailablePercentage: 25, the third replica cannot be placed and the volume stays degraded indefinitely. Check nodes.longhorn.io scheduling and /runbooks/longhorn-capacity.

5. replicaAutoBalance: best-effort is shuffling replicas

Normal background behaviour, transient. If degradation follows a pattern of replicas moving and never settling, that is worth an operator's attention.

When to escalate or stop

• If more than one volume is degraded and I/O-PSI on any node is climbing, freeze rebuilds first and then get the operator. That combination is the 2026-09-07 failure mode in progress.
• Never reboot, cordon or re-provision a second node while any volume is degraded. This is the single most important rule in this runbook.
• Do not kubectl drain. This cluster deliberately does not drain. nodeDrainPolicy: block-if-contains-last-replica is now declared (2026-09-13; before that it was in force only as Longhorn's own default, declared nowhere), so Longhorn will BLOCK a drain that would remove a volume's last healthy replica — that is a backstop against the worst case, not permission to drain. The upgrade plan still rejects system-upgrade-controller because "its auto-drain fights this cluster's slow-HDD Longhorn", and the node-loss conformance test evicts only its own writer pod.
• Do not delete a degraded volume's replicas to "force a clean rebuild" unless the operator agrees — with 3 replicas and one already missing you are one mistake from /runbooks/longhorn-faulted.
• Escalate for: a node that will not come back, a disk replacement, or a volume degraded for more than a few hours with all three nodes Ready (that is a Longhorn bug, not an operational state).

---

Source of truth: runbooks/longhorn-degraded.md. Index: /runbooks/ Records: DECISIONS-LOG.md "2026-08-31 — Post-upgrade stabilization" (the reboot tax) and "2026-09-07 — k3s-prod-1 NotReady"; docs/plans/cluster-upgrade-plan.md §6.2/§7 (the freeze/thaw).