fix(longhorn): the I/O caps a runbook promised were never declared — declare them

`runbooks/longhorn-degraded.md` told operators that, after the 2026-09-07 etcd
I/O-starvation incident, `concurrentVolumeBackupRestorePerNodeLimit` and
`snapshotHeavyTaskConcurrentLimit` had been tightened 5 -> 1. Measured live on
prod today: both read **5**, and neither name appeared anywhere in this
repository — so nothing declared them and nothing would have restored them.
Fourth instance of config that looked applied and was not.

Landed in `platform/longhorn/helmrelease.yaml` `defaultSettings` (the path Flux
reconciles; `values.yaml` is the reference mirror), NOT by `kubectl patch` —
a hand-applied Setting is exactly how this gap arose and Helm would silently
rewrite it on its next upgrade.

  concurrentVolumeBackupRestorePerNodeLimit      5 -> 1
  snapshotHeavyTaskConcurrentLimit              5 -> 1
  replicaReplenishmentWaitInterval              600     PINNED, unchanged
  backupConcurrentLimit / restoreConcurrentLimit  2 / 2 PINNED, unchanged
  concurrentAutomaticEngineUpgradePerNodeLimit  0       PINNED, unchanged
  nodeDrainPolicy  block-if-contains-last-replica       PINNED — it was in
      force only as Longhorn's OWN DEFAULT, declared in no manifest and in no
      rendered ConfigMap, so an upstream default change would have moved it
      silently while two runbooks reasoned about drain safety from it.

EVERY KEY PROVEN TO LAND, not assumed. `helm template longhorn/longhorn
--version 1.12.1` (the version this chart range resolves to, per the
HelmRelease's own lastAttemptedRevision) on this branch's spec.values,
substituted as clusters/prod substitutes them, emits all eight into the rendered
`longhorn-default-setting` ConfigMap in kebab-case. Diffed against the LIVE
ConfigMap the change is purely ADDITIVE: 8 keys added, none altered, none lost.
The chain to the Settings is verified read-only on prod — all 115
settings.longhorn.io carry `longhorn.io/configmap-resource-version: 7917`
(= that ConfigMap's resourceVersion) with `status.applied: true`, and every key
in the ConfigMap equals its Setting's value. The two targets sit at 5 with the
key absent, i.e. Longhorn defaults, as measured.

`orphanAutoDeletion: true` REMOVED, verified inert two independent ways against
the deployed 1.12.1: the render emits no orphan key at all (Longhorn 1.9 renamed
it `orphanResourceAutoDeletion`, a list, and the chart drops unknown
defaultSettings keys with no error), and live, `setting/orphan-auto-deletion` is
NotFound while `orphan-resource-auto-deletion` reads "". Deliberately NOT
replaced — orphan deletion is extra writes on the constrained partition and buys
only space. Orphan cleanup stays manual by decision; the two runbooks that cite
the key by name are corrected rather than left to go stale.

BOTH NEW CAPS ARE 1, AND 1 COSTS SOMETHING — stated, not waved at.
`concurrentVolumeBackupRestorePerNodeLimit: 1` costs nothing TODAY: Longhorn has
no backup target (the `backup-target` Setting does not exist; still the
WS-STORAGE Phase-4 TODO), and velero's nightly run is not a Longhorn backup —
`snapshotVolumes: false` + `defaultVolumesToFsBackup: true` is kopia
file-system backup through velero's node-agent, which this Setting does not
gate; CNPG base backups are barman-level and ungated too. Its only consumer is
an R-STATE-03 restore, where serialising ~24 volumes beats losing etcd
mid-restore. Revisit when a Longhorn backup target lands.
`snapshotHeavyTaskConcurrentLimit: 1` rather than 2 because purge work is
generated by rebuild cleanup (auto-cleanup-system-generated-snapshot true, NO
RecurringJobs, 24 volumes / 24 snapshots) and rebuild is already serialised to
1 per node — so 1 keeps pace with its own generator, where 2 buys backlog drain
at the price of a second heavy writer during the exact rebuild that starved
etcd. The cost: space reclaims serially after a rebuild storm — real here (a
37.7 GiB snapshot once pinned a volume at 97.3% allocated), but instrumented by
LonghornDiskAllocationHigh and recoverable by hand, where a wedged etcd is not.
Headroom on the record: I/O PSI full is 0.30/0.62/0.37 % now, and 3.79 under a
full 18-replica rebuild after node 1's drive replacement, against a 5% warning
threshold. These caps are a BOUND, not a fix — etcd and Longhorn still share one
ext4 filesystem and nothing arbitrates the jbd2 journal fdatasync waits on.

NOT PORTED, deliberately: the same unmerged branch's node-level I/O arbitration
(`roles/etcd_io_priority` — BFQ on the shared device, nr_requests 64 -> 32,
systemd IOWeight drop-ins). The 2011 C400 it was written for was replaced on
2026-09-07 and the same load class measured 11.36 -> 3.79 I/O PSI with the node
Ready throughout; the role was never run against any cluster, its author
recorded it as unproven with a node-3-first sequenced rollout, and its own
README concedes it does not separate the two tenants. Node-wide blast radius,
low ceiling, stale measurements (node 1 reinstalled, founder moved to node 2).
It stays an operator option if I/O PSI climbs back toward 5% — which, unlike in
September, we would now be told about. Nothing under infra/ansible/ is touched.

ALSO ALREADY DONE, so skipped: that branch's NodeIOPressureHigh/Critical and the
dashboard I/O row are on main already (efc6db6) with runbooks/node-io.md.
Verified live, not by reading the file: both rules loaded in
platform-health.nodes, health ok, state inactive, and
ALERTS{alertname=~"NodeIOPressure.*"} returns no samples.

Runbooks corrected in both directions: longhorn-degraded.md's prominent "DO NOT
RELY ON THAT: it was never applied" block becomes an accurate statement of what
is now declared, what to verify before trusting it, what it cannot do, and what
it costs; longhorn-capacity.md / node-disk.md stop citing a key that no longer
exists; node-io.md gains the one line an operator needs mid-incident.

No new ${...} substitution variables and not one dollar sign added anywhere in
the diff, so neither StrictPostBuildSubstitutions nor the L0 doubled-dollar guard
is affected.

Co-Authored-By: Claude Opus 5 <[email protected]>

DECISIONS-LOG.md

@@ -3474,3 +3474,109 @@ volume on the cluster.
 after the 2026-09-07 incident. Read live on prod today: `snapshot-heavy-task-concurrent-limit` is
 **5**. Either the change never landed or it was reverted; the runbook is currently telling an operator
 a knob is set that is not. Out of scope for this branch.
+
+## 2026-09-13 — the Longhorn I/O caps a runbook promised are now DECLARED; the Ansible half is not ported (branch `fix/longhorn-io-caps-actually-applied`)
+
+Closes the "Also noticed, NOT changed" note directly above. `runbooks/longhorn-degraded.md` told
+operators that `concurrentVolumeBackupRestorePerNodeLimit` and `snapshotHeavyTaskConcurrentLimit` had
+been tightened 5 → 1 after the 2026-09-07 etcd-starvation incident. Measured live on prod, both read
+**5**, and neither name appeared anywhere in the repository — so nothing declared them and nothing
+would have restored them. This is the **fourth** instance of config that looked applied and was not
+(after `orphanAutoDeletion`, the missing Longhorn ServiceMonitor, and the inert app SLI rules).
+
+### What landed, declaratively
+
+`platform/longhorn/helmrelease.yaml` `defaultSettings` — the only path Flux reconciles for these, and
+`platform/longhorn/values.yaml` is the reference mirror:
+
+| setting | now | was | why |
+|---|---|---|---|
+| `concurrentVolumeBackupRestorePerNodeLimit` | **1** | 5 (undeclared default) | five parallel full-volume writes onto the partition etcd fsyncs to |
+| `snapshotHeavyTaskConcurrentLimit` | **1** | 5 (undeclared default) | purge/clone rewrite whole snapshot disk files |
+| `replicaReplenishmentWaitInterval` | 600 | 600 | PINNED, not changed |
+| `backupConcurrentLimit` / `restoreConcurrentLimit` | 2 / 2 | 2 / 2 | PINNED, not changed |
+| `concurrentAutomaticEngineUpgradePerNodeLimit` | 0 | 0 | PINNED; engine upgrade stays operator-driven |
+| `nodeDrainPolicy` | `block-if-contains-last-replica` | same, **but only as Longhorn's own default** | it was in force by coincidence, declared nowhere |
+| `orphanAutoDeletion: true` | **removed** | inert since 1.12 | renamed `orphanResourceAutoDeletion`; the chart drops unknown keys silently |
+
+**Every key was proven to LAND, not assumed.** `helm template longhorn/longhorn --version 1.12.1`
+(the version the chart range resolves to, per the HelmRelease's `lastAttemptedRevision`) on this
+branch's own `spec.values`, substituted as `clusters/prod` substitutes them, emits all eight into the
+rendered `longhorn-default-setting` ConfigMap in kebab-case. The diff against the LIVE ConfigMap is
+purely additive: 8 keys added, none changed, none removed — which also confirms that removing
+`orphanAutoDeletion` changes the render by nothing at all, because it was never in it.
+
+**The chain to the live Settings, verified read-only on prod:** all 115 `settings.longhorn.io` objects
+carry `longhorn.io/configmap-resource-version: 7917` (= the ConfigMap's current `resourceVersion`)
+with `status.applied: true`, and every key currently in the ConfigMap equals its Setting's value. So
+the ConfigMap is authoritative and the controller re-applies on each revision. The two target
+Settings sit at 5 with the key absent from the ConfigMap — they are Longhorn defaults, exactly as
+measured. The final flip to 1/1 happens when this branch merges and Flux reconciles; it was
+deliberately NOT hand-applied, because a `kubectl` edit is how this gap arose, and a hand-applied
+ConfigMap would be silently rewritten by Helm's next upgrade.
+
+### The 1-vs-2 judgement, and what 1 costs
+
+Both went to **1**, for different reasons, and the cost is stated rather than waved at.
+
+* `concurrentVolumeBackupRestorePerNodeLimit: 1` **costs nothing today.** Longhorn has no backup
+  target configured (the `backup-target` Setting does not exist on the live cluster; it is still the
+  WS-STORAGE Phase-4 TODO), and velero's nightly schedule is **not** a Longhorn backup — it runs
+  `snapshotVolumes: false` with `defaultVolumesToFsBackup: true`, i.e. kopia file-system backup
+  through velero's node-agent, which this Setting does not gate. CNPG base backups are
+  barman/Postgres-level and are not gated either. The only consumer is an R-STATE-03 restore, where
+  serialising ~24 volumes is the right trade against losing etcd mid-restore. **Revisit when a
+  Longhorn backup target lands** — from that day it starts costing recovery time.
+* `snapshotHeavyTaskConcurrentLimit: 1`, **not 2.** The generator of purge work here is rebuild
+  cleanup (`auto-cleanup-system-generated-snapshot` is true, there are **no** RecurringJobs, 24
+  volumes hold 24 snapshots), and rebuild is already serialised to 1 per node — so purge at 1 keeps
+  pace with its own generator, while 2 would only drain an existing backlog faster at the price of a
+  second heavy writer during precisely the rebuild that starved etcd. **The cost is real and is
+  recorded in the runbooks:** space is reclaimed serially after a rebuild storm, the failure mode
+  that already produced a 37.7 GiB snapshot pinning a volume at 97.3% allocated. It is instrumented
+  (`LonghornDiskAllocationHigh` / `LonghornDiskUnschedulable`) and recoverable by hand; a wedged etcd
+  is neither.
+* Headroom, so the size of the bet is on the record: I/O PSI full on the three nodes right now is
+  0.30 / 0.62 / 0.37 %, and the worst measured value under a full 18-replica rebuild after node 1's
+  drive replacement is **3.79** against the 5% `NodeIOPressureHigh` threshold. About 1.2 points. That
+  is why a second concurrent heavy writer is worth refusing and why none of this is called a fix.
+
+### The Ansible half (`etcd_io_priority`) is NOT ported — the premise moved
+
+`fix/etcd-io-priority` also carried `roles/etcd_io_priority` + `playbooks/etcd-io-priority.yml`: BFQ
+on the shared device, `nr_requests` 64 → 32, `IOWeight=10000` drop-ins on `system.slice` and
+`k3s.service`, all read back and asserted. Assessed and declined:
+
+1. **The drive it was written for is gone.** §0c: node 1's 2011 C400 was replaced on 2026-09-07 and
+   the same load class measured I/O PSI full **11.36 → 3.79**, node Ready throughout — into node 3's
+   band (2.47). The branch's own §2 note concluded the shared-device premise was "right about node 2
+   and wrong about node 1"; the node-1-shaped part of that premise no longer exists.
+2. **It was never run against any cluster**, and its author recorded it as *unproven on this
+   hardware* — "it fails loudly if the settings do not land, it cannot tell you they helped" — with a
+   sequenced rollout (node 3 first, `--limit`, a day of `NodeIOPressureHigh` observation). That is an
+   operator procedure, not a merge.
+3. **Its own README states it does not separate the tenants.** Both still write one ext4 filesystem
+   and no cgroup or scheduler control arbitrates the jbd2 journal `fdatasync` waits on. Low ceiling,
+   node-wide blast radius: swapping the elevator and halving `nr_requests` on the live device holding
+   etcd's WAL and 24 attached volumes' replicas.
+4. **Its measurements are stale by construction.** Node 1 was reinstalled after the drive swap and
+   the founder moved to node 2; the containerd-ppid, cgroup-path and `io.latency`-absence findings all
+   need re-verifying on today's nodes before the asserts mean anything.
+
+**Recommendation:** keep it as a named operator option to revisit *if* I/O PSI climbs back toward 5%
+— which, unlike in September, we would now be told about. The real fix remains §2's exit condition:
+give etcd its own device. Nothing under `infra/ansible/` is touched by this branch.
+
+No repo-level constraint blocking `infra/ansible` commits was found (searched `ci/`, `conformance/`,
+`.gitea/`, and the log): the nearest real mechanism is `ci/touched-requirements.sh`'s path map, where
+`infra/*` attributes **R-ENV-01**, pulling the `dev-nodes-provisioned-as-code` L0 check — which drives
+`ansible-playbook --syntax-check`/`--list-hosts` over the **dev** provisioning chain — into the gate
+for any Ansible edit. Recorded because that is a real cost of touching `infra/ansible` now that dev is
+decommissioned, and it is a second reason the port would not have been cheap.
+
+### The I/O-pressure alert was already ported — skipped, verified
+
+`cb28fc8`'s `NodeIOPressureHigh` / `NodeIOPressureCritical` and the dashboard I/O row are already on
+`main` (as `efc6db6`), with `runbooks/node-io.md` alongside them. Verified live rather than by reading
+the file: both rules are loaded in the `platform-health.nodes` group, `health: ok`, `state: inactive`,
+and `ALERTS{alertname=~"NodeIOPressure.*"}` returns no samples. Nothing to port, no test added.