home-network-k3s-impl/runbooks/longhorn-degraded.md

longhorn-degraded — A Longhorn volume has been degraded for 10 minutes

Alert: LonghornVolumeDegraded (warning, 10m) File: platform/monitoring/alerts.yaml, group platform-health.longhorn Expression: longhorn_volume_robustness{state="degraded"} == 1 (one-hot over state, NOT a 0-3 enum)

What this means here

A volume has fewer healthy replicas than it should. It is still serving I/O — that is the difference between degraded and faulted (/runbooks/longhorn-faulted). What you have lost is redundancy, and on a 3-node cluster with hard node anti-affinity (replicaSoftAntiAffinity: false, numberOfReplicas: 3) one degraded volume means one more node loss away from data loss.

Degraded is the NORMAL state after any node reboot on this cluster, and it is expensive. Recorded plainly in the stabilization notes: "Longhorn on the HDD: every node reboot degrades all 18 volumes → full rebuild (slow; 30%+ IO-delay). Roll reboots one node at a time; wait robustness=healthy before the next." The rebuild is the thing that hurts, not the degradation: rebuild writes land on the same partition as the embedded etcd WAL (/data/k3s and /data/longhorn share one device), and that is the mechanism behind the 2026-09-07 k3s-prod-1 NotReady outage. So the question to answer is never just "why is it degraded" but "is the rebuild about to take a node out". Cross-read /runbooks/node-io.

Verify the alert can fire (measured 2026-09-13, and it was inert TWICE)

This rule was inert for two independent reasons, both fixed on 2026-09-13, and either one alone was enough to make a faulted or degraded volume alert nobody:

1. No scrape. count(longhorn_volume_robustness) returned 0 series — the longhorn_* family was absent from the TSDB because the chart's metrics.serviceMonitor.enabled defaulted false. Fixed in platform/longhorn/helmrelease.yaml.
2. Wrong metric shape. With the series present the rule STILL could not fire. It read longhorn_volume_robustness == 2 on the assumption of a 0–3 enum; on longhorn 1.12.1 the metric is one-hot over a state label with values 0/1 only. Measured on prod the same day: count(longhorn_volume_robustness) = 96 (24 volumes × 4 states), count by (state) (longhorn_volume_robustness == 1) = {state="healthy"} 24, and count(longhorn_volume_robustness == 2) = empty. Fixed by keying on the label.

Both directions are now pinned offline by platform/monitoring/tests/longhorn.test.yaml, which keeps a regression case asserting the old == 2 form matches nothing. Check the live series before trusting this alert's silence:

infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \
  prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
  /bin/promtool query instant http://localhost:9090 \
  'count by (state) (longhorn_volume_robustness)'

Four states, 24 each, is healthy wiring. Empty means the scrape has broken and this alert is inert again — do not infer "no alert, so volumes are fine".

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two dev.sh invocations at once; they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git (git push origin main is the deploy). Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

The truth about robustness, straight from the CRs (this works whether or not the metric does):

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
     -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,STATE:.status.state,NODE:.status.currentNodeID,SIZE:.spec.size'

Which replicas are missing, and where:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system get replicas.longhorn.io \
     -o custom-columns=NAME:.metadata.name,VOL:.spec.volumeName,NODE:.spec.nodeID,RUNNING:.status.currentState | sort -k2'

Which PVC/workload is affected:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get pv -o custom-columns=PV:.metadata.name,CLAIM:.spec.claimRef.namespace/.spec.claimRef.name,SC:.spec.storageClassName'

Are the nodes and their disks schedulable, and is a rebuild in flight?

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o wide; echo; \
   kubectl --context k3s-prod -n longhorn-system get settings.longhorn.io concurrent-replica-rebuild-per-node-limit -o custom-columns=NAME:.metadata.name,VALUE:.value'

And the pressure that rebuild is causing — the reason this alert matters:

ssh k3s-prod-1 'cat /proc/pressure/io'

Likely causes, most likely first

1. A node rebooted or was briefly NotReady (overwhelmingly the most common)

Expected, and it self-heals by rebuilding. Your job is to keep the rebuild from taking a node down. Remediation (operator, mutating) — freeze rebuilds while replicas will reattach (an in-place bounce):

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system patch settings.longhorn.io \
     concurrent-replica-rebuild-per-node-limit --type=merge -p "{\"value\":\"0\"}"'

Thaw to "1" — never higher — once every volume reads healthy. Use 1 rather than 0 when the replicas are genuinely gone (an OS re-provision), so replenishment can actually proceed. concurrentReplicaRebuildPerNodeLimit: 1 was already in force during the 2026-09-07 outage and held; do not "fix" this by setting a knob that is already set. replicaRebuildingBandwidthLimit is v2/SPDK-only and does nothing here; rebuildConcurrentSyncLimit is already at its floor of 1. What was actually tightened after that incident was every other Longhorn I/O source: concurrentVolumeBackupRestorePerNodeLimit and snapshotHeavyTaskConcurrentLimit 5 → 1.

2. A node is still down

Then the rebuild cannot complete and this will not clear. Go to /runbooks/node-down first; there is nothing to fix in Longhorn.

3. A disk was replaced and Longhorn refuses it

Signature: DiskFilesystemChanged — "record diskUUID doesn't match the one on the disk". Remediation, in exactly this order (the validating webhook enforces it): disable the disk → remove the disk entry → re-add it. Seen during the 2026-09-07 drive swap.

4. No node can hold the replica (anti-affinity or space)

Hard anti-affinity means a 3-replica volume needs three distinct schedulable nodes. With one node cordoned or its disk over storageMinimalAvailablePercentage: 25, the third replica cannot be placed and the volume stays degraded indefinitely. Check nodes.longhorn.io scheduling and /runbooks/longhorn-capacity.

5. replicaAutoBalance: best-effort is shuffling replicas

Normal background behaviour, transient. If degradation follows a pattern of replicas moving and never settling, that is worth an operator's attention.

When to escalate or stop

• If more than one volume is degraded and I/O-PSI on any node is climbing, freeze rebuilds first and then get the operator. That combination is the 2026-09-07 failure mode in progress.
• Never reboot, cordon or re-provision a second node while any volume is degraded. This is the single most important rule in this runbook.
• Do not kubectl drain. This cluster deliberately does not drain: there is no Longhorn nodeDrainPolicy configured anywhere in the repo, the upgrade plan rejects system-upgrade-controller because "its auto-drain fights this cluster's slow-HDD Longhorn", and the node-loss conformance test evicts only its own writer pod.
• Do not delete a degraded volume's replicas to "force a clean rebuild" unless the operator agrees — with 3 replicas and one already missing you are one mistake from /runbooks/longhorn-faulted.
• Escalate for: a node that will not come back, a disk replacement, or a volume degraded for more than a few hours with all three nodes Ready (that is a Longhorn bug, not an operational state).

---

Source of truth: runbooks/longhorn-degraded.md. Index: /runbooks/ Records: DECISIONS-LOG.md "2026-08-31 — Post-upgrade stabilization" (the reboot tax) and "2026-09-07 — k3s-prod-1 NotReady"; docs/plans/cluster-upgrade-plan.md §6.2/§7 (the freeze/thaw).