fix(longhorn): the I/O caps a runbook promised were never declared — declare them

`runbooks/longhorn-degraded.md` told operators that, after the 2026-09-07 etcd
I/O-starvation incident, `concurrentVolumeBackupRestorePerNodeLimit` and
`snapshotHeavyTaskConcurrentLimit` had been tightened 5 -> 1. Measured live on
prod today: both read **5**, and neither name appeared anywhere in this
repository — so nothing declared them and nothing would have restored them.
Fourth instance of config that looked applied and was not.

Landed in `platform/longhorn/helmrelease.yaml` `defaultSettings` (the path Flux
reconciles; `values.yaml` is the reference mirror), NOT by `kubectl patch` —
a hand-applied Setting is exactly how this gap arose and Helm would silently
rewrite it on its next upgrade.

  concurrentVolumeBackupRestorePerNodeLimit      5 -> 1
  snapshotHeavyTaskConcurrentLimit              5 -> 1
  replicaReplenishmentWaitInterval              600     PINNED, unchanged
  backupConcurrentLimit / restoreConcurrentLimit  2 / 2 PINNED, unchanged
  concurrentAutomaticEngineUpgradePerNodeLimit  0       PINNED, unchanged
  nodeDrainPolicy  block-if-contains-last-replica       PINNED — it was in
      force only as Longhorn's OWN DEFAULT, declared in no manifest and in no
      rendered ConfigMap, so an upstream default change would have moved it
      silently while two runbooks reasoned about drain safety from it.

EVERY KEY PROVEN TO LAND, not assumed. `helm template longhorn/longhorn
--version 1.12.1` (the version this chart range resolves to, per the
HelmRelease's own lastAttemptedRevision) on this branch's spec.values,
substituted as clusters/prod substitutes them, emits all eight into the rendered
`longhorn-default-setting` ConfigMap in kebab-case. Diffed against the LIVE
ConfigMap the change is purely ADDITIVE: 8 keys added, none altered, none lost.
The chain to the Settings is verified read-only on prod — all 115
settings.longhorn.io carry `longhorn.io/configmap-resource-version: 7917`
(= that ConfigMap's resourceVersion) with `status.applied: true`, and every key
in the ConfigMap equals its Setting's value. The two targets sit at 5 with the
key absent, i.e. Longhorn defaults, as measured.

`orphanAutoDeletion: true` REMOVED, verified inert two independent ways against
the deployed 1.12.1: the render emits no orphan key at all (Longhorn 1.9 renamed
it `orphanResourceAutoDeletion`, a list, and the chart drops unknown
defaultSettings keys with no error), and live, `setting/orphan-auto-deletion` is
NotFound while `orphan-resource-auto-deletion` reads "". Deliberately NOT
replaced — orphan deletion is extra writes on the constrained partition and buys
only space. Orphan cleanup stays manual by decision; the two runbooks that cite
the key by name are corrected rather than left to go stale.

BOTH NEW CAPS ARE 1, AND 1 COSTS SOMETHING — stated, not waved at.
`concurrentVolumeBackupRestorePerNodeLimit: 1` costs nothing TODAY: Longhorn has
no backup target (the `backup-target` Setting does not exist; still the
WS-STORAGE Phase-4 TODO), and velero's nightly run is not a Longhorn backup —
`snapshotVolumes: false` + `defaultVolumesToFsBackup: true` is kopia
file-system backup through velero's node-agent, which this Setting does not
gate; CNPG base backups are barman-level and ungated too. Its only consumer is
an R-STATE-03 restore, where serialising ~24 volumes beats losing etcd
mid-restore. Revisit when a Longhorn backup target lands.
`snapshotHeavyTaskConcurrentLimit: 1` rather than 2 because purge work is
generated by rebuild cleanup (auto-cleanup-system-generated-snapshot true, NO
RecurringJobs, 24 volumes / 24 snapshots) and rebuild is already serialised to
1 per node — so 1 keeps pace with its own generator, where 2 buys backlog drain
at the price of a second heavy writer during the exact rebuild that starved
etcd. The cost: space reclaims serially after a rebuild storm — real here (a
37.7 GiB snapshot once pinned a volume at 97.3% allocated), but instrumented by
LonghornDiskAllocationHigh and recoverable by hand, where a wedged etcd is not.
Headroom on the record: I/O PSI full is 0.30/0.62/0.37 % now, and 3.79 under a
full 18-replica rebuild after node 1's drive replacement, against a 5% warning
threshold. These caps are a BOUND, not a fix — etcd and Longhorn still share one
ext4 filesystem and nothing arbitrates the jbd2 journal fdatasync waits on.

NOT PORTED, deliberately: the same unmerged branch's node-level I/O arbitration
(`roles/etcd_io_priority` — BFQ on the shared device, nr_requests 64 -> 32,
systemd IOWeight drop-ins). The 2011 C400 it was written for was replaced on
2026-09-07 and the same load class measured 11.36 -> 3.79 I/O PSI with the node
Ready throughout; the role was never run against any cluster, its author
recorded it as unproven with a node-3-first sequenced rollout, and its own
README concedes it does not separate the two tenants. Node-wide blast radius,
low ceiling, stale measurements (node 1 reinstalled, founder moved to node 2).
It stays an operator option if I/O PSI climbs back toward 5% — which, unlike in
September, we would now be told about. Nothing under infra/ansible/ is touched.

ALSO ALREADY DONE, so skipped: that branch's NodeIOPressureHigh/Critical and the
dashboard I/O row are on main already (efc6db6) with runbooks/node-io.md.
Verified live, not by reading the file: both rules loaded in
platform-health.nodes, health ok, state inactive, and
ALERTS{alertname=~"NodeIOPressure.*"} returns no samples.

Runbooks corrected in both directions: longhorn-degraded.md's prominent "DO NOT
RELY ON THAT: it was never applied" block becomes an accurate statement of what
is now declared, what to verify before trusting it, what it cannot do, and what
it costs; longhorn-capacity.md / node-disk.md stop citing a key that no longer
exists; node-io.md gains the one line an operator needs mid-incident.

No new ${...} substitution variables and not one dollar sign added anywhere in
the diff, so neither StrictPostBuildSubstitutions nor the L0 doubled-dollar guard
is affected.

Co-Authored-By: Claude Opus 5 <[email protected]>

runbooks/longhorn-capacity.md

@@ -163,11 +163,17 @@ disk-full, ending in a postmaster PANIC. Go to `/runbooks/cnpg-archiving`. **Nev
 hand.**
 
 ### 2. Snapshots and orphaned replicas accumulating
-Longhorn snapshots are not free, and cleanup here is **manual**: `orphanAutoDeletion: true` in
-`platform/longhorn/helmrelease.yaml` has been **silently inert since Longhorn 1.12** — it was renamed
-`orphanResourceAutoDeletion` (now a list) and the chart drops unknown `defaultSettings` keys with no
-error. Verify the key's current state in that file before citing it; then prune orphans and stale
-snapshots from the Longhorn UI.
+Longhorn snapshots are not free, and cleanup here is **manual**. `orphanAutoDeletion: true` used to
+sit in `platform/longhorn/helmrelease.yaml` and had been **silently inert since Longhorn 1.12** — it
+was renamed `orphanResourceAutoDeletion` (now a list) and the chart drops unknown `defaultSettings`
+keys with no error. It was **removed on 2026-09-13** and deliberately not replaced, so orphan cleanup
+is manual by decision, not by accident: prune orphans and stale snapshots from the Longhorn UI.
+
+Note also that **snapshot purge and clone now run one at a time per node**
+(`snapshotHeavyTaskConcurrentLimit: 1`, declared 2026-09-13 to keep Longhorn from starving etcd's WAL
+fsync). Reclaim after a rebuild storm is therefore *serial and slower by design*. If allocation is
+climbing and purge is the bottleneck, the operator may raise the Setting for the duration and watch
+`NodeIOPressureHigh` — see `runbooks/longhorn-degraded.md` for the tradeoff as recorded.
 
 ### 3. A workload genuinely grew
 Plex/Sonarr/Radarr **media** does not live on Longhorn — it is static NFS from
@@ -199,5 +205,6 @@ answer (a ~51 GB hot tier: Prometheus, CNPG, Loki, Gitea). Operator planning wor
 
 ---
 Source of truth: `runbooks/longhorn-capacity.md`. Index: `/runbooks/`
-Records: `platform/longhorn/helmrelease.yaml` (the settings above and the inert `orphanAutoDeletion`);
+Records: `platform/longhorn/helmrelease.yaml` (the settings above, the I/O throttle block, and the
+removal of the inert `orphanAutoDeletion`);
 `platform/cnpg/alerts.yaml` header (the WAL disk-full clock); `docs/BULK-NFS-STORAGE.md`.

runbooks/longhorn-degraded.md

@@ -133,25 +133,58 @@ Thaw to `"1"` — never higher — once every volume reads `healthy`. Use `1` ra
 replicas are genuinely gone (an OS re-provision), so replenishment can actually proceed.
 `concurrentReplicaRebuildPerNodeLimit: 1` was **already in force during the 2026-09-07 outage and
 held**; do not "fix" this by setting a knob that is already set. `replicaRebuildingBandwidthLimit` is
-**v2/SPDK-only** and does nothing here; `rebuildConcurrentSyncLimit` is already at its floor of 1.
-What was *intended* after that incident was to tighten every **other** Longhorn I/O source —
-`concurrentVolumeBackupRestorePerNodeLimit` and `snapshotHeavyTaskConcurrentLimit` 5 -> 1.
+**v2/SPDK-only** and does nothing here, and there is no `rebuild-concurrent-sync-limit` Setting on
+1.12.1 at all. What the incident actually called for was tightening every **other** Longhorn I/O
+source — and as of 2026-09-13 that is declared in `platform/longhorn/helmrelease.yaml`
+(`defaultSettings`), not just intended:
 
-> **DO NOT RELY ON THAT: it was never applied.** Measured on prod 2026-09-13,
-> `snapshot-heavy-task-concurrent-limit` = **5** and
-> `concurrent-volume-backup-restore-per-node-limit` = **5**. Neither name appears anywhere in this
-> repository, so nothing declares them and nothing would restore them. The work lives on the
-> UNMERGED branch `fix/etcd-io-priority` ("cap every OTHER I/O source, and delete one setting that
-> was inert"), whose tree is far behind `main` and must be cherry-picked, not merged.
+| setting | declared | was, before |
+|---|---|---|
+| `concurrent-replica-rebuild-per-node-limit` | 1 | 1 (already declared) |
+| `concurrent-volume-backup-restore-per-node-limit` | **1** | 5 (Longhorn default, undeclared) |
+| `snapshot-heavy-task-concurrent-limit` | **1** | 5 (Longhorn default, undeclared) |
+| `replica-replenishment-wait-interval` | 600 | 600 (pinned, not changed) |
+| `backup-concurrent-limit` / `restore-concurrent-limit` | 2 / 2 | 2 / 2 (pinned) |
+| `concurrent-automatic-engine-upgrade-per-node-limit` | 0 | 0 (pinned) |
+| `node-drain-policy` | `block-if-contains-last-replica` | same, but as Longhorn's default only |
+
+> **WHAT CHANGED, AND WHAT TO CHECK BEFORE TRUSTING IT.** Until 2026-09-13 this page asserted a cap
+> that did not exist: `snapshot-heavy-task-concurrent-limit` and
+> `concurrent-volume-backup-restore-per-node-limit` both measured **5** on prod, neither name
+> appeared anywhere in the repository, and the work sat on an unmerged branch. They are now declared
+> in the HelmRelease and reconciled by Flux, which is what makes them survive a rebuild — a
+> `kubectl patch` is how the gap arose in the first place.
 >
-> Only two of the four mitigations this page implies are real:
-> `concurrent-replica-rebuild-per-node-limit` = **1** (its floor, as stated) and
-> `node-drain-policy` = **block-if-contains-last-replica**. Verify any limit before you trust it:
-> `kubectl -n longhorn-system get setting <name> -o jsonpath='{.value}'`.
+> **Verify, do not assume** (the whole point of this note):
+> `kubectl -n longhorn-system get setting <name> -o jsonpath='{.value}'`. If a value reads 5, the
+> HelmRelease has not reconciled — check `kubectl -n longhorn-system get helmrelease longhorn` and
+> that `longhorn-default-setting` carries the key
+> (`kubectl -n longhorn-system get cm longhorn-default-setting -o jsonpath='{.data}'`). The chart
+> **drops unknown `defaultSettings` keys with no error**, so "the key is in git" is not the same as
+> "the key is in effect" — `orphanAutoDeletion` was inert here for two minor versions for exactly
+> that reason, and has now been removed.
 >
-> This matters here specifically because the incident being described is etcd I/O starvation on a
-> node whose WAL shares a device with Longhorn. If you are reading this page during that failure,
-> the I/O cap you are counting on is not in place.
+> **These caps are a bound, not a fix.** They limit how much competing I/O Longhorn is *allowed* to
+> generate. etcd's WAL and every replica still share one partition on every prod node, and no
+> scheduler or cgroup control arbitrates the ext4 journal that `fdatasync` waits on. The measurable
+> improvement came from hardware: node 1's 2011 SATA SSD was replaced, and under the same load class
+> (an 18-replica rebuild with node 1 as the write target) I/O PSI full fell **11.36 -> 3.79** and the
+> node stayed Ready. 3.79 against the 5% `NodeIOPressureHigh` threshold is the headroom you have.
+>
+> **Cost of the two caps, so nobody is surprised during an incident:** snapshot purge and clone run
+> one at a time per node, so space is reclaimed serially after a rebuild storm (watch
+> `LonghornDiskAllocationHigh`); and a Longhorn *volume* restore runs one at a time per node. The
+> latter costs nothing today — no Longhorn backup target is configured, and velero's nightly run is
+> kopia file-system backup (`defaultVolumesToFsBackup: true`, `snapshotVolumes: false`), which this
+> setting does not gate. During a real DR restore an operator may raise
+> `concurrent-volume-backup-restore-per-node-limit` for the duration and watch
+> `NodeIOPressureHigh` — that is a deliberate, temporary patch, like the rebuild freeze above.
+>
+> **Not ported, deliberately:** the same unmerged branch also carried node-level I/O arbitration
+> (BFQ on the shared device, `nr_requests` 64 -> 32, systemd `IOWeight` drop-ins;
+> `infra/ansible/roles/etcd_io_priority`). It was never run against any cluster, its own author
+> recorded it as unproven and node-3-first, and the drive it was written for has since been replaced.
+> It remains an operator option if I/O PSI climbs back toward 5%, not a merged default.
 
 ### 2. A node is still down
 Then the rebuild cannot complete and this will not clear. Go to `/runbooks/node-down` first; there is
@@ -178,10 +211,12 @@ settling, that is worth an operator's attention.
   and then get the operator.** That combination is the 2026-09-07 failure mode in progress.
 - **Never reboot, cordon or re-provision a second node while any volume is degraded.** This is the
   single most important rule in this runbook.
-- **Do not `kubectl drain`.** This cluster deliberately does not drain: there is no Longhorn
-  `nodeDrainPolicy` configured anywhere in the repo, the upgrade plan rejects
-  system-upgrade-controller because "its auto-drain fights this cluster's slow-HDD Longhorn", and the
-  node-loss conformance test evicts only its own writer pod.
+- **Do not `kubectl drain`.** This cluster deliberately does not drain. `nodeDrainPolicy:
+  block-if-contains-last-replica` is now declared (2026-09-13; before that it was in force only as
+  Longhorn's own default, declared nowhere), so Longhorn will BLOCK a drain that would remove a
+  volume's last healthy replica — that is a backstop against the worst case, not permission to drain.
+  The upgrade plan still rejects system-upgrade-controller because "its auto-drain fights this
+  cluster's slow-HDD Longhorn", and the node-loss conformance test evicts only its own writer pod.
 - **Do not delete a degraded volume's replicas to "force a clean rebuild"** unless the operator agrees
   — with 3 replicas and one already missing you are one mistake from `/runbooks/longhorn-faulted`.
 - Escalate for: a node that will not come back, a disk replacement, or a volume degraded for more than

runbooks/node-disk.md

@@ -87,11 +87,12 @@ ssh k3s-prod-1 'sudo journalctl --disk-usage; sudo crictl images | wc -l; sudo d
 Hard node anti-affinity (`replicaSoftAntiAffinity: false`) means one replica per node per volume, but
 `replicaAutoBalance: best-effort` moves them. Prod sits around 269 GiB scheduled against an 829 GiB
 ceiling, so this is rarely tight — but a rebuild storm plus snapshots can spike it.
-Remediation: check for orphaned replicas and stale snapshots in the Longhorn UI. Note
-`orphanAutoDeletion: true` in `platform/longhorn/helmrelease.yaml` **has been silently inert since
-Longhorn 1.12** — it was renamed `orphanResourceAutoDeletion` (a list) and the chart drops unknown
-`defaultSettings` keys with no error. So orphan cleanup is manual. Verify the key's current state in
-the file before you trust it.
+Remediation: check for orphaned replicas and stale snapshots in the Longhorn UI. Orphan cleanup is
+**manual**: `orphanAutoDeletion: true` sat in `platform/longhorn/helmrelease.yaml` and had been
+**silently inert since Longhorn 1.12** — renamed `orphanResourceAutoDeletion` (a list), and the chart
+drops unknown `defaultSettings` keys with no error. Removed 2026-09-13 and deliberately not replaced.
+Snapshot purge also runs one at a time per node now (`snapshotHeavyTaskConcurrentLimit: 1`, the etcd
+WAL protection), so reclaim after a rebuild is serial by design.
 
 ### 2. A CNPG database is not recycling WAL
 The single most dangerous cause, because it is unbounded. With `archive_mode=on` and a failing

runbooks/node-io.md

@@ -122,7 +122,12 @@ infra/devcontainer/dev.sh run bash -c \
 Thaw to `"1"` — never higher — once all volumes read `healthy`. Use `0` when replicas will
 **reattach** (an in-place bounce); use `1` when they are genuinely gone (an OS re-provision).
 `concurrentReplicaRebuildPerNodeLimit: 1` was **already in force during the 2026-09-07 outage and
-held** — do not "fix" this by setting a knob that is already set.
+held** — do not "fix" this by setting a knob that is already set. Longhorn's *other* write sources
+are capped as of 2026-09-13 (`snapshot-heavy-task-concurrent-limit` and
+`concurrent-volume-backup-restore-per-node-limit`, both 5 -> 1, declared in
+`platform/longhorn/helmrelease.yaml`), so purge/clone and volume restore can no longer run five wide
+per node while a rebuild is in progress. Confirm rather than assume:
+`kubectl -n longhorn-system get setting <name> -o jsonpath='{.value}'`.
 
 ### 2. The node's disk is failing
 `k3s-prod-1`'s original 2011 `C400-MTFDDAC512MAM` had **61,440 reallocated sectors**, 31% endurance