home-network-k3s-impl/runbooks/longhorn-degraded.md

# longhorn-degraded — A Longhorn volume has been degraded for 10 minutes

**Alert:** `LonghornVolumeDegraded` (warning, 10m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.longhorn`
**Expression:** `longhorn_volume_robustness == 2` (0=unknown 1=healthy 2=degraded 3=faulted)

## What this means here

A volume has fewer healthy replicas than it should. It is **still serving I/O** — that is the
difference between degraded and faulted (`/runbooks/longhorn-faulted`). What you have lost is
redundancy, and on a 3-node cluster with hard node anti-affinity
(`replicaSoftAntiAffinity: false`, `numberOfReplicas: 3`) one degraded volume means one more node
loss away from data loss.

**Degraded is the NORMAL state after any node reboot on this cluster, and it is expensive.**
Recorded plainly in the stabilization notes: *"Longhorn on the HDD: every node reboot degrades all 18
volumes → full rebuild (slow; 30%+ IO-delay). Roll reboots one node at a time; wait
`robustness=healthy` before the next."* The rebuild is the thing that hurts, not the degradation:
rebuild writes land on the same partition as the embedded etcd WAL (`/data/k3s` and `/data/longhorn`
share one device), and that is the mechanism behind the 2026-09-07 `k3s-prod-1` NotReady outage. So
the question to answer is never just "why is it degraded" but "is the rebuild about to take a node
out". Cross-read `/runbooks/node-io`.
### Verify the alert can even fire (measured 2026-09-13)

Prometheus on prod scrapes 19 jobs: `apiserver`, `cdi-prometheus-metrics`, `coredns`,
`keycloak-discovery`, `keycloak-service`, `keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`,
`kube-prometheus-stack-grafana`, `kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`,
`kube-state-metrics`, `kubelet`, `kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`,
`node-exporter`, `oauth2-proxy`, `semaphore-db/semaphore-db`, `velero`, `velero/node-agent`.
There is **no** Longhorn, cert-manager, Flux-controller or Hubble/Cilium job. Check the series this
rule depends on before you trust its silence:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & \
   sleep 5; curl -s -G --data-urlencode "query=count by (job) (up)" localhost:19090/api/v1/query'
```

**This alert is currently INERT.** `count(longhorn_volume_robustness)` returned **0 series** on prod
on 2026-09-13: the whole `platform-health.longhorn` rule group has no data, because Longhorn ships no
ServiceMonitor here. That is a separate open defect, and the practical consequence is that **you will
probably learn about a degraded volume from an application symptom or from `volumes.longhorn.io`, not
from this alert**. Do not infer "no alert, so volumes are fine".
## Running the commands in this runbook

`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.

## First commands

The truth about robustness, straight from the CRs (this works whether or not the metric does):

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
     -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,STATE:.status.state,NODE:.status.currentNodeID,SIZE:.spec.size'
```

Which replicas are missing, and where:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system get replicas.longhorn.io \
     -o custom-columns=NAME:.metadata.name,VOL:.spec.volumeName,NODE:.spec.nodeID,RUNNING:.status.currentState | sort -k2'
```

Which PVC/workload is affected:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get pv -o custom-columns=PV:.metadata.name,CLAIM:.spec.claimRef.namespace/.spec.claimRef.name,SC:.spec.storageClassName'
```

Are the nodes and their disks schedulable, and is a rebuild in flight?

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o wide; echo; \
   kubectl --context k3s-prod -n longhorn-system get settings.longhorn.io concurrent-replica-rebuild-per-node-limit -o custom-columns=NAME:.metadata.name,VALUE:.value'
```

And the pressure that rebuild is causing — the reason this alert matters:

```
ssh k3s-prod-1 'cat /proc/pressure/io'
```

## Likely causes, most likely first

### 1. A node rebooted or was briefly NotReady (overwhelmingly the most common)
Expected, and it self-heals by rebuilding. Your job is to keep the rebuild from taking a node down.
Remediation (operator, mutating) — freeze rebuilds while replicas will **reattach** (an in-place
bounce):

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system patch settings.longhorn.io \
     concurrent-replica-rebuild-per-node-limit --type=merge -p "{\"value\":\"0\"}"'
```

Thaw to `"1"` — never higher — once every volume reads `healthy`. Use `1` rather than `0` when the
replicas are genuinely gone (an OS re-provision), so replenishment can actually proceed.
`concurrentReplicaRebuildPerNodeLimit: 1` was **already in force during the 2026-09-07 outage and
held**; do not "fix" this by setting a knob that is already set. `replicaRebuildingBandwidthLimit` is
**v2/SPDK-only** and does nothing here; `rebuildConcurrentSyncLimit` is already at its floor of 1.
What was actually tightened after that incident was every *other* Longhorn I/O source:
`concurrentVolumeBackupRestorePerNodeLimit` and `snapshotHeavyTaskConcurrentLimit` 5 → 1.

### 2. A node is still down
Then the rebuild cannot complete and this will not clear. Go to `/runbooks/node-down` first; there is
nothing to fix in Longhorn.

### 3. A disk was replaced and Longhorn refuses it
Signature: `DiskFilesystemChanged` — *"record diskUUID doesn't match the one on the disk"*.
Remediation, in exactly this order (the validating webhook enforces it): **disable the disk → remove
the disk entry → re-add it**. Seen during the 2026-09-07 drive swap.

### 4. No node can hold the replica (anti-affinity or space)
Hard anti-affinity means a 3-replica volume needs three distinct schedulable nodes. With one node
cordoned or its disk over `storageMinimalAvailablePercentage: 25`, the third replica cannot be
placed and the volume stays degraded indefinitely. Check `nodes.longhorn.io` scheduling and
`/runbooks/longhorn-capacity`.

### 5. `replicaAutoBalance: best-effort` is shuffling replicas
Normal background behaviour, transient. If degradation follows a pattern of replicas moving and never
settling, that is worth an operator's attention.

## When to escalate or stop

- **If more than one volume is degraded and I/O-PSI on any node is climbing, freeze rebuilds first
  and then get the operator.** That combination is the 2026-09-07 failure mode in progress.
- **Never reboot, cordon or re-provision a second node while any volume is degraded.** This is the
  single most important rule in this runbook.
- **Do not `kubectl drain`.** This cluster deliberately does not drain: there is no Longhorn
  `nodeDrainPolicy` configured anywhere in the repo, the upgrade plan rejects
  system-upgrade-controller because "its auto-drain fights this cluster's slow-HDD Longhorn", and the
  node-loss conformance test evicts only its own writer pod.
- **Do not delete a degraded volume's replicas to "force a clean rebuild"** unless the operator agrees
  — with 3 replicas and one already missing you are one mistake from `/runbooks/longhorn-faulted`.
- Escalate for: a node that will not come back, a disk replacement, or a volume degraded for more than
  a few hours with all three nodes Ready (that is a Longhorn bug, not an operational state).

---
Source of truth: `runbooks/longhorn-degraded.md`. Index: `/runbooks/`
Records: `DECISIONS-LOG.md` "2026-08-31 — Post-upgrade stabilization" (the reboot tax) and
"2026-09-07 — k3s-prod-1 NotReady"; `docs/plans/cluster-upgrade-plan.md` §6.2/§7 (the freeze/thaw).