jakman85andClaude Opus 5
8e017fad91
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
231 lines
14 KiB
Markdown
231 lines
14 KiB
Markdown
# longhorn-degraded — A Longhorn volume has been degraded for 10 minutes
|
||
|
||
**Alert:** `LonghornVolumeDegraded` (warning, 10m)
|
||
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.longhorn`
|
||
**Expression:** `longhorn_volume_robustness{state="degraded"} == 1` (one-hot over `state`, NOT a 0-3 enum)
|
||
|
||
## What this means here
|
||
|
||
A volume has fewer healthy replicas than it should. It is **still serving I/O** — that is the
|
||
difference between degraded and faulted (`/runbooks/longhorn-faulted`). What you have lost is
|
||
redundancy, and on a 3-node cluster with hard node anti-affinity
|
||
(`replicaSoftAntiAffinity: false`, `numberOfReplicas: 3`) one degraded volume means one more node
|
||
loss away from data loss.
|
||
|
||
**Degraded is the NORMAL state after any node reboot on this cluster, and it is expensive.**
|
||
Recorded plainly in the stabilization notes: *"Longhorn on the HDD: every node reboot degrades all 18
|
||
volumes → full rebuild (slow; 30%+ IO-delay). Roll reboots one node at a time; wait
|
||
`robustness=healthy` before the next."* The rebuild is the thing that hurts, not the degradation:
|
||
rebuild writes land on the same partition as the embedded etcd WAL (`/data/k3s` and `/data/longhorn`
|
||
share one device), and that is the mechanism behind the 2026-09-07 `k3s-prod-1` NotReady outage. So
|
||
the question to answer is never just "why is it degraded" but "is the rebuild about to take a node
|
||
out". Cross-read `/runbooks/node-io`.
|
||
### Verify the alert can fire (measured 2026-09-13, and it was inert TWICE)
|
||
|
||
This rule was inert for two independent reasons, both fixed on 2026-09-13, and either one alone was
|
||
enough to make a faulted or degraded volume alert nobody:
|
||
|
||
1. **No scrape.** `count(longhorn_volume_robustness)` returned 0 series — the longhorn_* family was
|
||
absent from the TSDB because the chart's `metrics.serviceMonitor.enabled` defaulted false. Fixed
|
||
in `platform/longhorn/helmrelease.yaml`.
|
||
2. **Wrong metric shape.** With the series present the rule STILL could not fire. It read
|
||
`longhorn_volume_robustness == 2` on the assumption of a 0–3 enum; on longhorn 1.12.1 the metric
|
||
is one-hot over a `state` label with values 0/1 only. Measured on prod the same day:
|
||
`count(longhorn_volume_robustness)` = 96 (24 volumes × 4 states),
|
||
`count by (state) (longhorn_volume_robustness == 1)` = `{state="healthy"} 24`, and
|
||
`count(longhorn_volume_robustness == 2)` = **empty**. Fixed by keying on the label.
|
||
|
||
Both directions are now pinned offline by `platform/monitoring/tests/longhorn.test.yaml`, which keeps
|
||
a regression case asserting the old `== 2` form matches nothing. Check the live series before
|
||
trusting this alert's silence:
|
||
|
||
```
|
||
infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \
|
||
prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
|
||
/bin/promtool query instant http://localhost:9090 \
|
||
'count by (state) (longhorn_volume_robustness)'
|
||
```
|
||
|
||
Four states, 24 each, is healthy wiring. Empty means the scrape has broken and this alert is inert
|
||
again — do not infer "no alert, so volumes are fine".
|
||
## Running the commands in this runbook
|
||
|
||
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
||
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
||
that hang is not the cluster being down. Every command below runs as:
|
||
|
||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
||
|
||
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
||
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
|
||
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
|
||
the same thing next month as it does today; one that relies on the default does not, and the failure is
|
||
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
|
||
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
|
||
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
|
||
interfere.
|
||
|
||
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
||
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
||
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
||
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
||
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
||
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||
|
||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||
and it says so.
|
||
|
||
## First commands
|
||
|
||
The truth about robustness, straight from the CRs (this works whether or not the metric does):
|
||
|
||
```
|
||
infra/devcontainer/dev.sh run bash -c \
|
||
'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
|
||
-o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,STATE:.status.state,NODE:.status.currentNodeID,SIZE:.spec.size'
|
||
```
|
||
|
||
Which replicas are missing, and where:
|
||
|
||
```
|
||
infra/devcontainer/dev.sh run bash -c \
|
||
'kubectl --context k3s-prod -n longhorn-system get replicas.longhorn.io \
|
||
-o custom-columns=NAME:.metadata.name,VOL:.spec.volumeName,NODE:.spec.nodeID,RUNNING:.status.currentState | sort -k2'
|
||
```
|
||
|
||
Which PVC/workload is affected:
|
||
|
||
```
|
||
infra/devcontainer/dev.sh run bash -c \
|
||
'kubectl --context k3s-prod get pv -o custom-columns=PV:.metadata.name,CLAIM:.spec.claimRef.namespace/.spec.claimRef.name,SC:.spec.storageClassName'
|
||
```
|
||
|
||
Are the nodes and their disks schedulable, and is a rebuild in flight?
|
||
|
||
```
|
||
infra/devcontainer/dev.sh run bash -c \
|
||
'kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o wide; echo; \
|
||
kubectl --context k3s-prod -n longhorn-system get settings.longhorn.io concurrent-replica-rebuild-per-node-limit -o custom-columns=NAME:.metadata.name,VALUE:.value'
|
||
```
|
||
|
||
And the pressure that rebuild is causing — the reason this alert matters:
|
||
|
||
```
|
||
ssh k3s-prod-1 'cat /proc/pressure/io'
|
||
```
|
||
|
||
## Likely causes, most likely first
|
||
|
||
### 1. A node rebooted or was briefly NotReady (overwhelmingly the most common)
|
||
Expected, and it self-heals by rebuilding. Your job is to keep the rebuild from taking a node down.
|
||
Remediation (operator, mutating) — freeze rebuilds while replicas will **reattach** (an in-place
|
||
bounce):
|
||
|
||
```
|
||
infra/devcontainer/dev.sh run bash -c \
|
||
'kubectl --context k3s-prod -n longhorn-system patch settings.longhorn.io \
|
||
concurrent-replica-rebuild-per-node-limit --type=merge -p "{\"value\":\"0\"}"'
|
||
```
|
||
|
||
Thaw to `"1"` — never higher — once every volume reads `healthy`. Use `1` rather than `0` when the
|
||
replicas are genuinely gone (an OS re-provision), so replenishment can actually proceed.
|
||
`concurrentReplicaRebuildPerNodeLimit: 1` was **already in force during the 2026-09-07 outage and
|
||
held**; do not "fix" this by setting a knob that is already set. `replicaRebuildingBandwidthLimit` is
|
||
**v2/SPDK-only** and does nothing here, and there is no `rebuild-concurrent-sync-limit` Setting on
|
||
1.12.1 at all. What the incident actually called for was tightening every **other** Longhorn I/O
|
||
source — and as of 2026-09-13 that is declared in `platform/longhorn/helmrelease.yaml`
|
||
(`defaultSettings`), not just intended:
|
||
|
||
| setting | declared | was, before |
|
||
|---|---|---|
|
||
| `concurrent-replica-rebuild-per-node-limit` | 1 | 1 (already declared) |
|
||
| `concurrent-volume-backup-restore-per-node-limit` | **1** | 5 (Longhorn default, undeclared) |
|
||
| `snapshot-heavy-task-concurrent-limit` | **1** | 5 (Longhorn default, undeclared) |
|
||
| `replica-replenishment-wait-interval` | 600 | 600 (pinned, not changed) |
|
||
| `backup-concurrent-limit` / `restore-concurrent-limit` | 2 / 2 | 2 / 2 (pinned) |
|
||
| `concurrent-automatic-engine-upgrade-per-node-limit` | 0 | 0 (pinned) |
|
||
| `node-drain-policy` | `block-if-contains-last-replica` | same, but as Longhorn's default only |
|
||
|
||
> **WHAT CHANGED, AND WHAT TO CHECK BEFORE TRUSTING IT.** Until 2026-09-13 this page asserted a cap
|
||
> that did not exist: `snapshot-heavy-task-concurrent-limit` and
|
||
> `concurrent-volume-backup-restore-per-node-limit` both measured **5** on prod, neither name
|
||
> appeared anywhere in the repository, and the work sat on an unmerged branch. They are now declared
|
||
> in the HelmRelease and reconciled by Flux, which is what makes them survive a rebuild — a
|
||
> `kubectl patch` is how the gap arose in the first place.
|
||
>
|
||
> **Verify, do not assume** (the whole point of this note):
|
||
> `kubectl -n longhorn-system get setting <name> -o jsonpath='{.value}'`. If a value reads 5, the
|
||
> HelmRelease has not reconciled — check `kubectl -n longhorn-system get helmrelease longhorn` and
|
||
> that `longhorn-default-setting` carries the key
|
||
> (`kubectl -n longhorn-system get cm longhorn-default-setting -o jsonpath='{.data}'`). The chart
|
||
> **drops unknown `defaultSettings` keys with no error**, so "the key is in git" is not the same as
|
||
> "the key is in effect" — `orphanAutoDeletion` was inert here for two minor versions for exactly
|
||
> that reason, and has now been removed.
|
||
>
|
||
> **These caps are a bound, not a fix.** They limit how much competing I/O Longhorn is *allowed* to
|
||
> generate. etcd's WAL and every replica still share one partition on every prod node, and no
|
||
> scheduler or cgroup control arbitrates the ext4 journal that `fdatasync` waits on. The measurable
|
||
> improvement came from hardware: node 1's 2011 SATA SSD was replaced, and under the same load class
|
||
> (an 18-replica rebuild with node 1 as the write target) I/O PSI full fell **11.36 -> 3.79** and the
|
||
> node stayed Ready. 3.79 against the 5% `NodeIOPressureHigh` threshold is the headroom you have.
|
||
>
|
||
> **Cost of the two caps, so nobody is surprised during an incident:** snapshot purge and clone run
|
||
> one at a time per node, so space is reclaimed serially after a rebuild storm (watch
|
||
> `LonghornDiskAllocationHigh`); and a Longhorn *volume* restore runs one at a time per node. The
|
||
> latter costs nothing today — no Longhorn backup target is configured, and velero's nightly run is
|
||
> kopia file-system backup (`defaultVolumesToFsBackup: true`, `snapshotVolumes: false`), which this
|
||
> setting does not gate. During a real DR restore an operator may raise
|
||
> `concurrent-volume-backup-restore-per-node-limit` for the duration and watch
|
||
> `NodeIOPressureHigh` — that is a deliberate, temporary patch, like the rebuild freeze above.
|
||
>
|
||
> **Not ported, deliberately:** the same unmerged branch also carried node-level I/O arbitration
|
||
> (BFQ on the shared device, `nr_requests` 64 -> 32, systemd `IOWeight` drop-ins;
|
||
> `infra/ansible/roles/etcd_io_priority`). It was never run against any cluster, its own author
|
||
> recorded it as unproven and node-3-first, and the drive it was written for has since been replaced.
|
||
> It remains an operator option if I/O PSI climbs back toward 5%, not a merged default.
|
||
|
||
### 2. A node is still down
|
||
Then the rebuild cannot complete and this will not clear. Go to `/runbooks/node-down` first; there is
|
||
nothing to fix in Longhorn.
|
||
|
||
### 3. A disk was replaced and Longhorn refuses it
|
||
Signature: `DiskFilesystemChanged` — *"record diskUUID doesn't match the one on the disk"*.
|
||
Remediation, in exactly this order (the validating webhook enforces it): **disable the disk → remove
|
||
the disk entry → re-add it**. Seen during the 2026-09-07 drive swap.
|
||
|
||
### 4. No node can hold the replica (anti-affinity or space)
|
||
Hard anti-affinity means a 3-replica volume needs three distinct schedulable nodes. With one node
|
||
cordoned or its disk over `storageMinimalAvailablePercentage: 25`, the third replica cannot be
|
||
placed and the volume stays degraded indefinitely. Check `nodes.longhorn.io` scheduling and
|
||
`/runbooks/longhorn-capacity`.
|
||
|
||
### 5. `replicaAutoBalance: best-effort` is shuffling replicas
|
||
Normal background behaviour, transient. If degradation follows a pattern of replicas moving and never
|
||
settling, that is worth an operator's attention.
|
||
|
||
## When to escalate or stop
|
||
|
||
- **If more than one volume is degraded and I/O-PSI on any node is climbing, freeze rebuilds first
|
||
and then get the operator.** That combination is the 2026-09-07 failure mode in progress.
|
||
- **Never reboot, cordon or re-provision a second node while any volume is degraded.** This is the
|
||
single most important rule in this runbook.
|
||
- **Do not `kubectl drain`.** This cluster deliberately does not drain. `nodeDrainPolicy:
|
||
block-if-contains-last-replica` is now declared (2026-09-13; before that it was in force only as
|
||
Longhorn's own default, declared nowhere), so Longhorn will BLOCK a drain that would remove a
|
||
volume's last healthy replica — that is a backstop against the worst case, not permission to drain.
|
||
The upgrade plan still rejects system-upgrade-controller because "its auto-drain fights this
|
||
cluster's slow-HDD Longhorn", and the node-loss conformance test evicts only its own writer pod.
|
||
- **Do not delete a degraded volume's replicas to "force a clean rebuild"** unless the operator agrees
|
||
— with 3 replicas and one already missing you are one mistake from `/runbooks/longhorn-faulted`.
|
||
- Escalate for: a node that will not come back, a disk replacement, or a volume degraded for more than
|
||
a few hours with all three nodes Ready (that is a Longhorn bug, not an operational state).
|
||
|
||
---
|
||
Source of truth: `runbooks/longhorn-degraded.md`. Index: `/runbooks/`
|
||
Records: `DECISIONS-LOG.md` "2026-08-31 — Post-upgrade stabilization" (the reboot tax) and
|
||
"2026-09-07 — k3s-prod-1 NotReady"; `docs/plans/cluster-upgrade-plan.md` §6.2/§7 (the freeze/thaw).
|