# longhorn-faulted — A Longhorn volume is FAULTED (data at risk, volume offline) **Alert:** `LonghornVolumeFaulted` (critical, 1m) **File:** `platform/monitoring/alerts.yaml`, group `platform-health.longhorn` **Expression:** `longhorn_volume_robustness{state="faulted"} == 1` (one-hot over `state`, NOT a 0-3 enum) ## What this means here Faulted means Longhorn has **no usable replica**. The volume is offline; the workload's writes are failing. This is the most serious storage state on the cluster and the `for: 1m` is deliberate — there is no benign transient version of this. Understand what is and is not recoverable before you touch anything: - The volumes on this cluster are 3-replica with **hard** node anti-affinity, so faulted normally means two or three replicas are gone at once — two nodes lost, or one node lost while another volume's rebuild had not finished, or a disk that ate its replica. - **Standbys and replicas are not backups.** Recorded verbatim for the databases and true for all of this: *"Standbys replicate corruption and deletions faithfully; only the Barman archive is a backup."* For a CNPG volume (`keycloak-db`, `nextcloud-db`, `semaphore-db`) the real recovery path is the object-store archive, not Longhorn. For everything else it is Velero. - If this is a **`longhorn-vm`** volume (RWX Block, `migratable: true`) the VM is down, not just the disk. If it is **`longhorn-scratch`** (single replica, CDI scratch space) losing it is expected and harmless — re-run the import. ### Verify the alert can fire (measured 2026-09-13, and it was inert TWICE) This rule was inert for two independent reasons, both fixed on 2026-09-13. Either alone meant a faulted volume — the most serious storage state on this cluster — alerted nobody, and announced itself only as an application outage: 1. **No scrape.** `count(longhorn_volume_robustness)` returned 0 series; the chart's `metrics.serviceMonitor.enabled` defaulted false. Fixed in `platform/longhorn/helmrelease.yaml`. 2. **Wrong metric shape.** With the series present the rule STILL could not fire. It read `longhorn_volume_robustness == 3` on the assumption of a 0–3 enum; on longhorn 1.12.1 the metric is one-hot over a `state` label with values 0/1 only. Measured on prod the same day: `count(longhorn_volume_robustness)` = 96 (24 volumes × 4 states) and `count(longhorn_volume_robustness == 3)` = **empty**. Fixed by keying on the label. Both directions are now pinned offline by `platform/monitoring/tests/longhorn.test.yaml`. Check the live series before trusting this alert's silence: ``` infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \ prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \ /bin/promtool query instant http://localhost:9090 \ 'count by (state) (longhorn_volume_robustness)' ``` Four states, one entry per volume each, is healthy wiring. Empty means the scrape has broken and this alert is inert again — treat `volumes.longhorn.io` as the source of truth, never this alert's silence. ## Running the commands in this runbook `kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write `backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for agents here: every durable change goes through git. **`main` is a protected branch — a direct push is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, and it says so. ## First commands Stop and take inventory before changing anything. Which volume, which workload, which replicas exist: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \ -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,STATE:.status.state,NODE:.status.currentNodeID | grep -Ev "healthy"' ``` ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n longhorn-system get replicas.longhorn.io \ -o custom-columns=NAME:.metadata.name,VOL:.spec.volumeName,NODE:.spec.nodeID,STATE:.status.currentState,FAILED:.spec.failedAt | sort -k2' ``` ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get pv -o custom-columns=PV:.metadata.name,CLAIM:.spec.claimRef.namespace/.spec.claimRef.name,SC:.spec.storageClassName | grep ' ``` What recovery material exists — **fully-qualified names, or you will be told there is nothing**: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get backups.velero.io -A; echo; \ kubectl --context k3s-prod get backups.postgresql.cnpg.io -A; echo; \ kubectl --context k3s-prod -n longhorn-system get backups.longhorn.io' ``` `kubectl get backup -A` prints "No resources found" on this cluster even when six Velero and 27 CNPG backups exist. Never make a recovery decision from the short name. Node and disk state: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get nodes; kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o wide' ``` ## Likely causes, most likely first ### 1. Two nodes were unavailable at once Usually a second reboot or cordon while the first node's rebuilds were still running. Remediation: bring the nodes back. Longhorn will reattach if the replica data survived on disk — which is why you must not delete replicas before checking. ### 2. A disk failed and took its replica with it `k3s-prod-1`'s original 2011 SSD had 61,440 reallocated sectors before it was replaced; the dev-era Seagate had 448 current-pending plus 448 offline-uncorrectable. Check SMART on the node holding the failed replica (`/runbooks/node-io` has the command). If the disk is bad, the replica is gone — the path forward is restore, not repair. ### 3. All replicas were evicted or deleted Check whether a disk was disabled, a node removed from Longhorn, or a `replicas.longhorn.io` object deleted. `staleReplicaTimeout: 30` (minutes) means Longhorn eventually discards replicas it considers stale. ### 4. It is a single-replica volume by design `longhorn-scratch` is `numberOfReplicas: 1`. Losing it is not an incident — delete the DataVolume and re-import. ## Remediation, in order 1. **Do not delete the volume, the PVC, or any replica.** A faulted volume with a stale replica is recoverable; a deleted one is not. 2. If a node is down, restore the node (`/runbooks/node-down`). That is the highest-probability fix. 3. If Longhorn shows a replica with data but marked failed, an operator can salvage it from the Longhorn UI (*Volume → Salvage*) — this is the documented path for "all replicas failed but the data is still there". 4. If the volume backs a **CNPG database**, do not try to salvage Postgres files. Recover the Cluster from the Barman archive (`/runbooks/cnpg-archiving` for how to confirm the archive is real first). 5. Otherwise restore from Velero. Note honestly: `dr-restore` is **authored and has never been executed** (`conformance.home.k3s/never-executed`), and the R2 restore path has not been rehearsed. Expect to be the first person to run it, and read `/runbooks/backup-failure` for the R2 plugin caveats before you rely on a backup's metadata. ## When to escalate or stop - **Get the operator now.** This is a critical, data-at-risk alert; it is not a runbook-and-carry-on situation. Everything above is diagnosis, and steps 3–5 are operator actions. - **Stop before any destructive command.** Specifically: no `delete pvc`, no `delete volumes.longhorn.io`, no `delete replicas.longhorn.io`, no touching files under `/data/longhorn`. - **Do not reboot anything** while a volume is faulted. - If the affected volume is `keycloak-db`, expect a full SSO outage as a knock-on: kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth all depend on it. The certificate-based kubeconfig is an independent break-glass path and still works. --- Source of truth: `runbooks/longhorn-faulted.md`. Index: `/runbooks/` Records: `DECISIONS-LOG.md` (drive failures 2026-08-22 and 2026-09-07; `dr-restore` never executed).