jakman85andClaude Opus 5
8e017fad91
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
174 lines
9.7 KiB
Markdown
174 lines
9.7 KiB
Markdown
# longhorn-faulted — A Longhorn volume is FAULTED (data at risk, volume offline)
|
||
|
||
**Alert:** `LonghornVolumeFaulted` (critical, 1m)
|
||
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.longhorn`
|
||
**Expression:** `longhorn_volume_robustness{state="faulted"} == 1` (one-hot over `state`, NOT a 0-3 enum)
|
||
|
||
## What this means here
|
||
|
||
Faulted means Longhorn has **no usable replica**. The volume is offline; the workload's writes are
|
||
failing. This is the most serious storage state on the cluster and the `for: 1m` is deliberate — there
|
||
is no benign transient version of this.
|
||
|
||
Understand what is and is not recoverable before you touch anything:
|
||
|
||
- The volumes on this cluster are 3-replica with **hard** node anti-affinity, so faulted normally
|
||
means two or three replicas are gone at once — two nodes lost, or one node lost while another
|
||
volume's rebuild had not finished, or a disk that ate its replica.
|
||
- **Standbys and replicas are not backups.** Recorded verbatim for the databases and true for all of
|
||
this: *"Standbys replicate corruption and deletions faithfully; only the Barman archive is a
|
||
backup."* For a CNPG volume (`keycloak-db`, `nextcloud-db`, `semaphore-db`) the real recovery path
|
||
is the object-store archive, not Longhorn. For everything else it is Velero.
|
||
- If this is a **`longhorn-vm`** volume (RWX Block, `migratable: true`) the VM is down, not just the
|
||
disk. If it is **`longhorn-scratch`** (single replica, CDI scratch space) losing it is expected and
|
||
harmless — re-run the import.
|
||
### Verify the alert can fire (measured 2026-09-13, and it was inert TWICE)
|
||
|
||
This rule was inert for two independent reasons, both fixed on 2026-09-13. Either alone meant a
|
||
faulted volume — the most serious storage state on this cluster — alerted nobody, and announced
|
||
itself only as an application outage:
|
||
|
||
1. **No scrape.** `count(longhorn_volume_robustness)` returned 0 series; the chart's
|
||
`metrics.serviceMonitor.enabled` defaulted false. Fixed in
|
||
`platform/longhorn/helmrelease.yaml`.
|
||
2. **Wrong metric shape.** With the series present the rule STILL could not fire. It read
|
||
`longhorn_volume_robustness == 3` on the assumption of a 0–3 enum; on longhorn 1.12.1 the metric
|
||
is one-hot over a `state` label with values 0/1 only. Measured on prod the same day:
|
||
`count(longhorn_volume_robustness)` = 96 (24 volumes × 4 states) and
|
||
`count(longhorn_volume_robustness == 3)` = **empty**. Fixed by keying on the label.
|
||
|
||
Both directions are now pinned offline by `platform/monitoring/tests/longhorn.test.yaml`. Check the
|
||
live series before trusting this alert's silence:
|
||
|
||
```
|
||
infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \
|
||
prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
|
||
/bin/promtool query instant http://localhost:9090 \
|
||
'count by (state) (longhorn_volume_robustness)'
|
||
```
|
||
|
||
Four states, one entry per volume each, is healthy wiring. Empty means the scrape has broken and this
|
||
alert is inert again — treat `volumes.longhorn.io` as the source of truth, never this alert's silence.
|
||
## Running the commands in this runbook
|
||
|
||
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
||
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
||
that hang is not the cluster being down. Every command below runs as:
|
||
|
||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
||
|
||
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
||
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
|
||
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
|
||
the same thing next month as it does today; one that relies on the default does not, and the failure is
|
||
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
|
||
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
|
||
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
|
||
interfere.
|
||
|
||
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
||
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
||
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
||
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
||
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
||
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||
|
||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||
and it says so.
|
||
|
||
## First commands
|
||
|
||
Stop and take inventory before changing anything. Which volume, which workload, which replicas exist:
|
||
|
||
```
|
||
infra/devcontainer/dev.sh run bash -c \
|
||
'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
|
||
-o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,STATE:.status.state,NODE:.status.currentNodeID | grep -Ev "healthy"'
|
||
```
|
||
|
||
```
|
||
infra/devcontainer/dev.sh run bash -c \
|
||
'kubectl --context k3s-prod -n longhorn-system get replicas.longhorn.io \
|
||
-o custom-columns=NAME:.metadata.name,VOL:.spec.volumeName,NODE:.spec.nodeID,STATE:.status.currentState,FAILED:.spec.failedAt | sort -k2'
|
||
```
|
||
|
||
```
|
||
infra/devcontainer/dev.sh run bash -c \
|
||
'kubectl --context k3s-prod get pv -o custom-columns=PV:.metadata.name,CLAIM:.spec.claimRef.namespace/.spec.claimRef.name,SC:.spec.storageClassName | grep <volume-name>'
|
||
```
|
||
|
||
What recovery material exists — **fully-qualified names, or you will be told there is nothing**:
|
||
|
||
```
|
||
infra/devcontainer/dev.sh run bash -c \
|
||
'kubectl --context k3s-prod get backups.velero.io -A; echo; \
|
||
kubectl --context k3s-prod get backups.postgresql.cnpg.io -A; echo; \
|
||
kubectl --context k3s-prod -n longhorn-system get backups.longhorn.io'
|
||
```
|
||
|
||
`kubectl get backup -A` prints "No resources found" on this cluster even when six Velero and 27 CNPG
|
||
backups exist. Never make a recovery decision from the short name.
|
||
|
||
Node and disk state:
|
||
|
||
```
|
||
infra/devcontainer/dev.sh run bash -c \
|
||
'kubectl --context k3s-prod get nodes; kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o wide'
|
||
```
|
||
|
||
## Likely causes, most likely first
|
||
|
||
### 1. Two nodes were unavailable at once
|
||
Usually a second reboot or cordon while the first node's rebuilds were still running. Remediation:
|
||
bring the nodes back. Longhorn will reattach if the replica data survived on disk — which is why you
|
||
must not delete replicas before checking.
|
||
|
||
### 2. A disk failed and took its replica with it
|
||
`k3s-prod-1`'s original 2011 SSD had 61,440 reallocated sectors before it was replaced; the dev-era
|
||
Seagate had 448 current-pending plus 448 offline-uncorrectable. Check SMART on the node holding the
|
||
failed replica (`/runbooks/node-io` has the command). If the disk is bad, the replica is gone — the
|
||
path forward is restore, not repair.
|
||
|
||
### 3. All replicas were evicted or deleted
|
||
Check whether a disk was disabled, a node removed from Longhorn, or a `replicas.longhorn.io` object
|
||
deleted. `staleReplicaTimeout: 30` (minutes) means Longhorn eventually discards replicas it considers
|
||
stale.
|
||
|
||
### 4. It is a single-replica volume by design
|
||
`longhorn-scratch` is `numberOfReplicas: 1`. Losing it is not an incident — delete the DataVolume and
|
||
re-import.
|
||
|
||
## Remediation, in order
|
||
|
||
1. **Do not delete the volume, the PVC, or any replica.** A faulted volume with a stale replica is
|
||
recoverable; a deleted one is not.
|
||
2. If a node is down, restore the node (`/runbooks/node-down`). That is the highest-probability fix.
|
||
3. If Longhorn shows a replica with data but marked failed, an operator can salvage it from the
|
||
Longhorn UI (*Volume → Salvage*) — this is the documented path for "all replicas failed but the
|
||
data is still there".
|
||
4. If the volume backs a **CNPG database**, do not try to salvage Postgres files. Recover the Cluster
|
||
from the Barman archive (`/runbooks/cnpg-archiving` for how to confirm the archive is real first).
|
||
5. Otherwise restore from Velero. Note honestly: `dr-restore` is **authored and has never been
|
||
executed** (`conformance.home.k3s/never-executed`), and the R2 restore path has not been rehearsed.
|
||
Expect to be the first person to run it, and read `/runbooks/backup-failure` for the R2 plugin
|
||
caveats before you rely on a backup's metadata.
|
||
|
||
## When to escalate or stop
|
||
|
||
- **Get the operator now.** This is a critical, data-at-risk alert; it is not a runbook-and-carry-on
|
||
situation. Everything above is diagnosis, and steps 3–5 are operator actions.
|
||
- **Stop before any destructive command.** Specifically: no `delete pvc`, no `delete
|
||
volumes.longhorn.io`, no `delete replicas.longhorn.io`, no touching files under `/data/longhorn`.
|
||
- **Do not reboot anything** while a volume is faulted.
|
||
- If the affected volume is `keycloak-db`, expect a full SSO outage as a knock-on: kubectl OIDC,
|
||
Grafana, Headlamp, Gitea and forward-auth all depend on it. The certificate-based kubeconfig is an
|
||
independent break-glass path and still works.
|
||
|
||
---
|
||
Source of truth: `runbooks/longhorn-faulted.md`. Index: `/runbooks/`
|
||
Records: `DECISIONS-LOG.md` (drive failures 2026-08-22 and 2026-09-07; `dr-restore` never executed).
|