fix(longhorn): the I/O caps a runbook promised were never declared — declare them

`runbooks/longhorn-degraded.md` told operators that, after the 2026-09-07 etcd
I/O-starvation incident, `concurrentVolumeBackupRestorePerNodeLimit` and
`snapshotHeavyTaskConcurrentLimit` had been tightened 5 -> 1. Measured live on
prod today: both read **5**, and neither name appeared anywhere in this
repository — so nothing declared them and nothing would have restored them.
Fourth instance of config that looked applied and was not.

Landed in `platform/longhorn/helmrelease.yaml` `defaultSettings` (the path Flux
reconciles; `values.yaml` is the reference mirror), NOT by `kubectl patch` —
a hand-applied Setting is exactly how this gap arose and Helm would silently
rewrite it on its next upgrade.

  concurrentVolumeBackupRestorePerNodeLimit      5 -> 1
  snapshotHeavyTaskConcurrentLimit              5 -> 1
  replicaReplenishmentWaitInterval              600     PINNED, unchanged
  backupConcurrentLimit / restoreConcurrentLimit  2 / 2 PINNED, unchanged
  concurrentAutomaticEngineUpgradePerNodeLimit  0       PINNED, unchanged
  nodeDrainPolicy  block-if-contains-last-replica       PINNED — it was in
      force only as Longhorn's OWN DEFAULT, declared in no manifest and in no
      rendered ConfigMap, so an upstream default change would have moved it
      silently while two runbooks reasoned about drain safety from it.

EVERY KEY PROVEN TO LAND, not assumed. `helm template longhorn/longhorn
--version 1.12.1` (the version this chart range resolves to, per the
HelmRelease's own lastAttemptedRevision) on this branch's spec.values,
substituted as clusters/prod substitutes them, emits all eight into the rendered
`longhorn-default-setting` ConfigMap in kebab-case. Diffed against the LIVE
ConfigMap the change is purely ADDITIVE: 8 keys added, none altered, none lost.
The chain to the Settings is verified read-only on prod — all 115
settings.longhorn.io carry `longhorn.io/configmap-resource-version: 7917`
(= that ConfigMap's resourceVersion) with `status.applied: true`, and every key
in the ConfigMap equals its Setting's value. The two targets sit at 5 with the
key absent, i.e. Longhorn defaults, as measured.

`orphanAutoDeletion: true` REMOVED, verified inert two independent ways against
the deployed 1.12.1: the render emits no orphan key at all (Longhorn 1.9 renamed
it `orphanResourceAutoDeletion`, a list, and the chart drops unknown
defaultSettings keys with no error), and live, `setting/orphan-auto-deletion` is
NotFound while `orphan-resource-auto-deletion` reads "". Deliberately NOT
replaced — orphan deletion is extra writes on the constrained partition and buys
only space. Orphan cleanup stays manual by decision; the two runbooks that cite
the key by name are corrected rather than left to go stale.

BOTH NEW CAPS ARE 1, AND 1 COSTS SOMETHING — stated, not waved at.
`concurrentVolumeBackupRestorePerNodeLimit: 1` costs nothing TODAY: Longhorn has
no backup target (the `backup-target` Setting does not exist; still the
WS-STORAGE Phase-4 TODO), and velero's nightly run is not a Longhorn backup —
`snapshotVolumes: false` + `defaultVolumesToFsBackup: true` is kopia
file-system backup through velero's node-agent, which this Setting does not
gate; CNPG base backups are barman-level and ungated too. Its only consumer is
an R-STATE-03 restore, where serialising ~24 volumes beats losing etcd
mid-restore. Revisit when a Longhorn backup target lands.
`snapshotHeavyTaskConcurrentLimit: 1` rather than 2 because purge work is
generated by rebuild cleanup (auto-cleanup-system-generated-snapshot true, NO
RecurringJobs, 24 volumes / 24 snapshots) and rebuild is already serialised to
1 per node — so 1 keeps pace with its own generator, where 2 buys backlog drain
at the price of a second heavy writer during the exact rebuild that starved
etcd. The cost: space reclaims serially after a rebuild storm — real here (a
37.7 GiB snapshot once pinned a volume at 97.3% allocated), but instrumented by
LonghornDiskAllocationHigh and recoverable by hand, where a wedged etcd is not.
Headroom on the record: I/O PSI full is 0.30/0.62/0.37 % now, and 3.79 under a
full 18-replica rebuild after node 1's drive replacement, against a 5% warning
threshold. These caps are a BOUND, not a fix — etcd and Longhorn still share one
ext4 filesystem and nothing arbitrates the jbd2 journal fdatasync waits on.

NOT PORTED, deliberately: the same unmerged branch's node-level I/O arbitration
(`roles/etcd_io_priority` — BFQ on the shared device, nr_requests 64 -> 32,
systemd IOWeight drop-ins). The 2011 C400 it was written for was replaced on
2026-09-07 and the same load class measured 11.36 -> 3.79 I/O PSI with the node
Ready throughout; the role was never run against any cluster, its author
recorded it as unproven with a node-3-first sequenced rollout, and its own
README concedes it does not separate the two tenants. Node-wide blast radius,
low ceiling, stale measurements (node 1 reinstalled, founder moved to node 2).
It stays an operator option if I/O PSI climbs back toward 5% — which, unlike in
September, we would now be told about. Nothing under infra/ansible/ is touched.

ALSO ALREADY DONE, so skipped: that branch's NodeIOPressureHigh/Critical and the
dashboard I/O row are on main already (efc6db6) with runbooks/node-io.md.
Verified live, not by reading the file: both rules loaded in
platform-health.nodes, health ok, state inactive, and
ALERTS{alertname=~"NodeIOPressure.*"} returns no samples.

Runbooks corrected in both directions: longhorn-degraded.md's prominent "DO NOT
RELY ON THAT: it was never applied" block becomes an accurate statement of what
is now declared, what to verify before trusting it, what it cannot do, and what
it costs; longhorn-capacity.md / node-disk.md stop citing a key that no longer
exists; node-io.md gains the one line an operator needs mid-incident.

No new ${...} substitution variables and not one dollar sign added anywhere in
the diff, so neither StrictPostBuildSubstitutions nor the L0 doubled-dollar guard
is affected.

Co-Authored-By: Claude Opus 5 <[email protected]>

platform/longhorn/helmrelease.yaml

@@ -76,15 +76,104 @@ spec:
       # never blocks scheduling).
       replicaAutoBalance: "best-effort"
 
-      # --- rebuild I/O throttle for slow disks (etcd WAL fsync protection) ---
-      # The k3s VM disks sit on HDDs, so a full-replica rebuild's sequential I/O can saturate the
-      # spindle and starve etcd's WAL fsync — which crash-loops etcd and flaps the kube-vip VIP.
-      # Cap Longhorn to rebuilding ONE replica per node at a time so a node bounce / replica
-      # replenishment can never storm the HDDs. Deliberately conservative for this slow-disk cluster
-      # (Longhorn's chart default is higher). Pair with replica-replenishment-wait-interval=600
-      # (Longhorn default) so brief node bounces don't trigger replenishment at all.
+      # --- I/O THROTTLES: bound every Longhorn write source that competes with etcd's WAL fsync ---
+      # WHY THIS BLOCK IS BIGGER THAN THE ONE LINE IT REPLACES. On 2026-09-07 k3s-prod-1 went
+      # NotReady ("Kubelet stopped posting node status") with the host up and SSH-able: its embedded
+      # etcd could not finish a WAL fdatasync (took=3.56s/6.71s/4.95s/3.40s against a 1 s
+      # expectation), "apply request took too long took=8.30s", then "etcdserver: too many
+      # requests". /proc/pressure/io FULL avg300 read 11.36 there against 0.75 (node 2, NVMe) and
+      # 1.29 (node 3). Cause: EVERY prod node puts /data/k3s (etcd WAL + containerd) and
+      # /data/longhorn (every replica) on ONE partition, and node 1's was a 2011 consumer SATA SSD
+      # with 61,440 reallocated sectors (infra/PROD-PROVISIONING.md §2 and §0c).
+      #
+      # THE HONEST HEADLINE: the rebuild path was ALREADY throttled and it is not what this block
+      # fixes. concurrentReplicaRebuildPerNodeLimit was 1 during that outage and it HELD — the 18
+      # volumes rebuilt one at a time — and the node stalled anyway. Longhorn offers no further
+      # rebuild lever on the v1 data engine: replicaRebuildingBandwidthLimit is V2/SPDK ONLY
+      # (verified live 2026-09-13, the Setting reads a v2-only map with no v1 key) and there is no
+      # rebuild-concurrent-sync-limit Setting on 1.12.1 at all (NotFound). What is bounded below is
+      # therefore every OTHER Longhorn write source, each of which measured 5 per node.
+      #
+      # WHAT THIS BLOCK CANNOT DO. Node 1's drive HAS since been replaced (§0c), and THAT is the
+      # real mitigation: under the same load class — an 18-replica rebuild with node 1 as the write
+      # target — IO PSI full fell 11.36 -> 3.79 and the node stayed Ready. These settings only bound
+      # how much competing I/O Longhorn is ALLOWED to generate. etcd and Longhorn still share one
+      # ext4 filesystem, and no scheduler or cgroup control arbitrates the jbd2 journal that
+      # fdatasync waits on. 3.79 against the 5% NodeIOPressureHigh threshold is about 1.2 points of
+      # headroom, which is exactly why a SECOND concurrent heavy writer is still worth refusing.
+      #
+      # EVERY KEY BELOW IS PROVEN TO LAND, not assumed. `helm template longhorn/longhorn --version
+      # 1.12.1` (the version this chart range resolves to, per the HelmRelease's own
+      # lastAttemptedRevision) emits each one into the rendered longhorn-default-setting ConfigMap
+      # in kebab-case, and a live Setting of that name exists for each. This matters because the
+      # chart DROPS unknown defaultSettings keys with no error — see the orphanAutoDeletion note
+      # further down for the setting that was inert here for two minor versions because nobody
+      # checked the render.
+
+      # Rebuild ONE replica per node at a time. Unchanged, and load-bearing: without it a node
+      # bounce replenishes every degraded volume at once. The freeze/thaw to "0"/"1" during an
+      # operator bounce (runbooks/longhorn-degraded.md, docs/plans/cluster-upgrade-plan.md) patches
+      # the Setting directly — deliberate and temporary; this line is the value it returns to.
       concurrentReplicaRebuildPerNodeLimit: 1
 
+      # Seconds Longhorn waits before replenishing a replica on a node that went away. This is
+      # Longhorn's own default and the live Setting already reads 600 — PINNED, not changed, so no
+      # future chart or manager default can silently shorten the window in which a node can be
+      # recovered by hand without triggering a rebuild storm.
+      replicaReplenishmentWaitInterval: 600
+
+      # WAS 5, MEASURED LIVE ON PROD 2026-09-13. Restores read from the backup target and write
+      # whole volumes to the local disk — the heaviest sustained write Longhorn can produce. 5 means
+      # five parallel full-volume writes onto the partition etcd is fsyncing to.
+      # THE COST OF 1, STATED HONESTLY: today it is zero. Longhorn has NO backup target configured
+      # (the backup-target Setting does not even exist on the live cluster — it is still the
+      # WS-STORAGE Phase 4 TODO at the foot of this file), and velero's nightly run is NOT a
+      # Longhorn backup: its schedules set snapshotVolumes:false with defaultVolumesToFsBackup:true,
+      # i.e. kopia file-system backup through velero's own node-agent, which this Setting does not
+      # gate. CNPG base backups are barman/Postgres-level and are not gated either. The only
+      # consumer is an R-STATE-03 DR restore of ~24 volumes, where serialising is the correct trade
+      # against losing etcd mid-restore — and an operator who needs restore throughput can raise the
+      # Setting for the duration and watch NodeIOPressureHigh. REVISIT when a Longhorn backup target
+      # actually lands: from that day this knob starts costing real recovery time.
+      concurrentVolumeBackupRestorePerNodeLimit: 1
+
+      # WAS 5, MEASURED LIVE ON PROD 2026-09-13. Snapshot "heavy tasks" are purge and clone: full
+      # reads and rewrites of snapshot disk files.
+      # WHY 1 AND NOT 2 — this balance was judged, not copied. The work that generates purges here
+      # is rebuild cleanup: auto-cleanup-system-generated-snapshot is true, there are NO
+      # RecurringJobs on this cluster, and the 24 volumes hold 24 snapshots between them. Purge
+      # therefore arrives at the rate rebuilds retire snapshots, which is ALREADY serialised to 1
+      # per node — so a purge concurrency of 1 keeps pace with its own generator. 2 would only drain
+      # an existing backlog faster, at the price of a second heavy writer on the disk during
+      # precisely the rebuild that starved etcd once, with about 1.2 points of measured PSI headroom
+      # to spend.
+      # THE COST OF 1: space is reclaimed serially after a rebuild storm. That is a REAL cost here —
+      # a 37.7 GiB system snapshot from prod-1's rebuild once pinned a volume at 97.3% ALLOCATED —
+      # but it is the cost that is already instrumented (LonghornDiskAllocationHigh,
+      # LonghornDiskUnschedulable) and recoverable by hand, where a wedged etcd is neither.
+      snapshotHeavyTaskConcurrentLimit: 1
+
+      # Worker threads WITHIN one backup / one restore. Both sit at Longhorn's default of 2; PINNED
+      # rather than lowered — combined with the per-node limit of 1 above, one volume at a time with
+      # two threads is already conservative, and pinning stops an upstream bump from widening it.
+      backupConcurrentLimit: 2
+      restoreConcurrentLimit: 2
+
+      # Automatic engine upgrades roll every volume's engine after a Longhorn minor bump — a
+      # cluster-wide restart-and-resync storm on the same shared disk. 0 disables it, making engine
+      # upgrade a deliberate operator action, which matches this HelmRelease's own
+      # remediateLastFailure:false posture on minor upgrades. It is Longhorn's default; PINNED so
+      # both halves of that one decision live in one file.
+      concurrentAutomaticEngineUpgradePerNodeLimit: 0
+
+      # Never drain a node that holds the last healthy replica of a volume. This was ALREADY the
+      # live value on 2026-09-13 — but only because it is Longhorn's own default: it appeared in NO
+      # manifest in this repo and in NO rendered ConfigMap, so an upstream default change would have
+      # moved it silently. PINNED, value unchanged. runbooks/longhorn-degraded.md and
+      # docs/plans/cluster-upgrade-plan.md both reason about drain safety, so it has to be a
+      # declaration rather than a coincidence.
+      nodeDrainPolicy: "block-if-contains-last-replica"
+
       # --- capacity guardrails (docs/04 "reserve disk headroom") ---
       # Reserve 20% of the dedicated data disk so a node never fills to 100% (which wedges the
       # kubelet and Longhorn alike). Dev disks are small, so headroom matters.
@@ -99,8 +188,23 @@ spec:
       # --- misc dev-appropriate defaults ---
       # Keep replicas spread for durability rather than co-locating with the pod.
       defaultDataLocality: "disabled"
-      # Reclaim replica space of deleted/failed volumes promptly in dev.
-      orphanAutoDeletion: true
+      # `orphanAutoDeletion: true` USED TO BE HERE AND WAS INERT. Removed 2026-09-13, having been
+      # verified dead two independent ways against the DEPLOYED 1.12.1:
+      #   1. RENDER: `helm template longhorn/longhorn --version 1.12.1` with that key set emits NO
+      #      orphan key whatsoever into longhorn-default-setting. Longhorn 1.9 replaced
+      #      orphanAutoDeletion (a bool) with orphanResourceAutoDeletion (a semicolon-separated list
+      #      such as "replica-data;instance"), and the chart template drops unknown defaultSettings
+      #      keys with no error.
+      #   2. LIVE, from the other end: the rendered longhorn-default-setting ConfigMap in
+      #      longhorn-system carries no orphan key at all, `get setting orphan-auto-deletion` is
+      #      NotFound, and `get setting orphan-resource-auto-deletion` reads "" — Longhorn's default.
+      # DELIBERATELY NOT REPLACED with orphanResourceAutoDeletion: orphan deletion is extra write
+      # work on the one partition that is this cluster's constraint (see the I/O block above) and it
+      # buys only reclaimed space, of which prod has hundreds of GB free. Leaving it unset keeps
+      # Longhorn's default of not auto-deleting, which is the conservative side here — orphan
+      # cleanup stays manual, as runbooks/longhorn-capacity.md and runbooks/node-disk.md say.
+      # Left as a comment rather than deleted silently, because "config that looks applied and is
+      # not" is the failure this repo keeps re-learning, and two runbooks cite this key by name.
 
     # --- SCRAPE THE STORAGE LAYER, or every Longhorn alert is INERT (added 2026-09-13) ----------
     # platform/monitoring/alerts.yaml ships the platform-health.longhorn group (as of 2026-09-13:

platform/longhorn/values.yaml

@@ -15,12 +15,30 @@
 #   replicaSoftAntiAffinity: false                   # HARD anti-affinity: one replica per node
 #   replicaZoneSoftAntiAffinity: true                # single zone (3 mini-PCs, no failure zones)
 #   replicaAutoBalance: best-effort                  # rebalance toward one-per-node when a node returns
-#   concurrentReplicaRebuildPerNodeLimit: 1          # throttle rebuild I/O so it can't starve etcd WAL fsync (HDD-backed)
+#
+#   # --- I/O throttles: keep Longhorn from starving etcd's WAL fsync. The etcd WAL and every
+#   # Longhorn replica share ONE partition on every prod node (PROD-PROVISIONING.md §2); see
+#   # helmrelease.yaml for the 2026-09-07 incident, the post-drive-replacement measurement, and the
+#   # cost of each value. The two marked "was 5" were measured at 5 on prod on 2026-09-13 while a
+#   # runbook claimed they were 1 — nothing in this repo declared them. ---
+#   concurrentReplicaRebuildPerNodeLimit: 1          # one replica rebuild per node at a time
+#   replicaReplenishmentWaitInterval: 600            # PINNED at Longhorn's default, not changed
+#   concurrentVolumeBackupRestorePerNodeLimit: 1     # was 5: five parallel full-volume restores
+#   snapshotHeavyTaskConcurrentLimit: 1              # was 5: snapshot purge/clone rewrites
+#   backupConcurrentLimit: 2                         # PINNED at Longhorn's default (threads per backup)
+#   restoreConcurrentLimit: 2                        # PINNED at Longhorn's default (threads per restore)
+#   concurrentAutomaticEngineUpgradePerNodeLimit: 0  # PINNED: engine upgrades are operator-driven
+#   nodeDrainPolicy: block-if-contains-last-replica  # PINNED: was live only as Longhorn's default
+#   # NOT AVAILABLE TO US, do not add: replicaRebuildingBandwidthLimit is v2-data-engine only, and
+#   # no rebuild-concurrent-sync-limit Setting exists on 1.12.1 at all.
+#
 #   storageReservedPercentageForDefaultDisk: "20"    # reserve disk headroom (docs/04 guardrail)
 #   storageMinimalAvailablePercentage: "25"          # stop scheduling below 25% free
 #   storageOverProvisioningPercentage: "100"         # dev: no thin over-provisioning (1:1)
 #   defaultDataLocality: disabled                    # keep replicas spread for durability
-#   orphanAutoDeletion: true                         # reclaim orphaned replica space promptly
+#   # orphanAutoDeletion — REMOVED 2026-09-13: the key does not exist in the 1.12 chart (renamed
+#   # orphanResourceAutoDeletion, now a list) and was being silently dropped from the render.
+#   # Deliberately NOT replaced; orphan cleanup stays manual. helmrelease.yaml carries the proof.
 #   backupTarget: <off-cluster>                      # TODO(WS-STORAGE, Phase 4): DR target (ADR-0003)
 #
 # metrics: