From f663e7c801bbfafd5465d33371046cea3899a414 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 13 Sep 2026 17:48:42 -0500 Subject: [PATCH] =?UTF-8?q?fix(longhorn):=20the=20I/O=20caps=20a=20runbook?= =?UTF-8?q?=20promised=20were=20never=20declared=20=E2=80=94=20declare=20t?= =?UTF-8?q?hem?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `runbooks/longhorn-degraded.md` told operators that, after the 2026-09-07 etcd I/O-starvation incident, `concurrentVolumeBackupRestorePerNodeLimit` and `snapshotHeavyTaskConcurrentLimit` had been tightened 5 -> 1. Measured live on prod today: both read **5**, and neither name appeared anywhere in this repository — so nothing declared them and nothing would have restored them. Fourth instance of config that looked applied and was not. Landed in `platform/longhorn/helmrelease.yaml` `defaultSettings` (the path Flux reconciles; `values.yaml` is the reference mirror), NOT by `kubectl patch` — a hand-applied Setting is exactly how this gap arose and Helm would silently rewrite it on its next upgrade. concurrentVolumeBackupRestorePerNodeLimit 5 -> 1 snapshotHeavyTaskConcurrentLimit 5 -> 1 replicaReplenishmentWaitInterval 600 PINNED, unchanged backupConcurrentLimit / restoreConcurrentLimit 2 / 2 PINNED, unchanged concurrentAutomaticEngineUpgradePerNodeLimit 0 PINNED, unchanged nodeDrainPolicy block-if-contains-last-replica PINNED — it was in force only as Longhorn's OWN DEFAULT, declared in no manifest and in no rendered ConfigMap, so an upstream default change would have moved it silently while two runbooks reasoned about drain safety from it. EVERY KEY PROVEN TO LAND, not assumed. `helm template longhorn/longhorn --version 1.12.1` (the version this chart range resolves to, per the HelmRelease's own lastAttemptedRevision) on this branch's spec.values, substituted as clusters/prod substitutes them, emits all eight into the rendered `longhorn-default-setting` ConfigMap in kebab-case. Diffed against the LIVE ConfigMap the change is purely ADDITIVE: 8 keys added, none altered, none lost. The chain to the Settings is verified read-only on prod — all 115 settings.longhorn.io carry `longhorn.io/configmap-resource-version: 7917` (= that ConfigMap's resourceVersion) with `status.applied: true`, and every key in the ConfigMap equals its Setting's value. The two targets sit at 5 with the key absent, i.e. Longhorn defaults, as measured. `orphanAutoDeletion: true` REMOVED, verified inert two independent ways against the deployed 1.12.1: the render emits no orphan key at all (Longhorn 1.9 renamed it `orphanResourceAutoDeletion`, a list, and the chart drops unknown defaultSettings keys with no error), and live, `setting/orphan-auto-deletion` is NotFound while `orphan-resource-auto-deletion` reads "". Deliberately NOT replaced — orphan deletion is extra writes on the constrained partition and buys only space. Orphan cleanup stays manual by decision; the two runbooks that cite the key by name are corrected rather than left to go stale. BOTH NEW CAPS ARE 1, AND 1 COSTS SOMETHING — stated, not waved at. `concurrentVolumeBackupRestorePerNodeLimit: 1` costs nothing TODAY: Longhorn has no backup target (the `backup-target` Setting does not exist; still the WS-STORAGE Phase-4 TODO), and velero's nightly run is not a Longhorn backup — `snapshotVolumes: false` + `defaultVolumesToFsBackup: true` is kopia file-system backup through velero's node-agent, which this Setting does not gate; CNPG base backups are barman-level and ungated too. Its only consumer is an R-STATE-03 restore, where serialising ~24 volumes beats losing etcd mid-restore. Revisit when a Longhorn backup target lands. `snapshotHeavyTaskConcurrentLimit: 1` rather than 2 because purge work is generated by rebuild cleanup (auto-cleanup-system-generated-snapshot true, NO RecurringJobs, 24 volumes / 24 snapshots) and rebuild is already serialised to 1 per node — so 1 keeps pace with its own generator, where 2 buys backlog drain at the price of a second heavy writer during the exact rebuild that starved etcd. The cost: space reclaims serially after a rebuild storm — real here (a 37.7 GiB snapshot once pinned a volume at 97.3% allocated), but instrumented by LonghornDiskAllocationHigh and recoverable by hand, where a wedged etcd is not. Headroom on the record: I/O PSI full is 0.30/0.62/0.37 % now, and 3.79 under a full 18-replica rebuild after node 1's drive replacement, against a 5% warning threshold. These caps are a BOUND, not a fix — etcd and Longhorn still share one ext4 filesystem and nothing arbitrates the jbd2 journal fdatasync waits on. NOT PORTED, deliberately: the same unmerged branch's node-level I/O arbitration (`roles/etcd_io_priority` — BFQ on the shared device, nr_requests 64 -> 32, systemd IOWeight drop-ins). The 2011 C400 it was written for was replaced on 2026-09-07 and the same load class measured 11.36 -> 3.79 I/O PSI with the node Ready throughout; the role was never run against any cluster, its author recorded it as unproven with a node-3-first sequenced rollout, and its own README concedes it does not separate the two tenants. Node-wide blast radius, low ceiling, stale measurements (node 1 reinstalled, founder moved to node 2). It stays an operator option if I/O PSI climbs back toward 5% — which, unlike in September, we would now be told about. Nothing under infra/ansible/ is touched. ALSO ALREADY DONE, so skipped: that branch's NodeIOPressureHigh/Critical and the dashboard I/O row are on main already (efc6db6) with runbooks/node-io.md. Verified live, not by reading the file: both rules loaded in platform-health.nodes, health ok, state inactive, and ALERTS{alertname=~"NodeIOPressure.*"} returns no samples. Runbooks corrected in both directions: longhorn-degraded.md's prominent "DO NOT RELY ON THAT: it was never applied" block becomes an accurate statement of what is now declared, what to verify before trusting it, what it cannot do, and what it costs; longhorn-capacity.md / node-disk.md stop citing a key that no longer exists; node-io.md gains the one line an operator needs mid-incident. No new ${...} substitution variables and not one dollar sign added anywhere in the diff, so neither StrictPostBuildSubstitutions nor the L0 doubled-dollar guard is affected. Co-Authored-By: Claude Opus 5 --- DECISIONS-LOG.md | 106 +++++++++++++++++++++++++ platform/longhorn/helmrelease.yaml | 122 ++++++++++++++++++++++++++--- platform/longhorn/values.yaml | 22 +++++- runbooks/longhorn-capacity.md | 19 +++-- runbooks/longhorn-degraded.md | 75 +++++++++++++----- runbooks/node-disk.md | 11 +-- runbooks/node-io.md | 7 +- 7 files changed, 319 insertions(+), 43 deletions(-) diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index b330e6b..d3b44b7 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -3474,3 +3474,109 @@ volume on the cluster. after the 2026-09-07 incident. Read live on prod today: `snapshot-heavy-task-concurrent-limit` is **5**. Either the change never landed or it was reverted; the runbook is currently telling an operator a knob is set that is not. Out of scope for this branch. + +## 2026-09-13 — the Longhorn I/O caps a runbook promised are now DECLARED; the Ansible half is not ported (branch `fix/longhorn-io-caps-actually-applied`) + +Closes the "Also noticed, NOT changed" note directly above. `runbooks/longhorn-degraded.md` told +operators that `concurrentVolumeBackupRestorePerNodeLimit` and `snapshotHeavyTaskConcurrentLimit` had +been tightened 5 → 1 after the 2026-09-07 etcd-starvation incident. Measured live on prod, both read +**5**, and neither name appeared anywhere in the repository — so nothing declared them and nothing +would have restored them. This is the **fourth** instance of config that looked applied and was not +(after `orphanAutoDeletion`, the missing Longhorn ServiceMonitor, and the inert app SLI rules). + +### What landed, declaratively + +`platform/longhorn/helmrelease.yaml` `defaultSettings` — the only path Flux reconciles for these, and +`platform/longhorn/values.yaml` is the reference mirror: + +| setting | now | was | why | +|---|---|---|---| +| `concurrentVolumeBackupRestorePerNodeLimit` | **1** | 5 (undeclared default) | five parallel full-volume writes onto the partition etcd fsyncs to | +| `snapshotHeavyTaskConcurrentLimit` | **1** | 5 (undeclared default) | purge/clone rewrite whole snapshot disk files | +| `replicaReplenishmentWaitInterval` | 600 | 600 | PINNED, not changed | +| `backupConcurrentLimit` / `restoreConcurrentLimit` | 2 / 2 | 2 / 2 | PINNED, not changed | +| `concurrentAutomaticEngineUpgradePerNodeLimit` | 0 | 0 | PINNED; engine upgrade stays operator-driven | +| `nodeDrainPolicy` | `block-if-contains-last-replica` | same, **but only as Longhorn's own default** | it was in force by coincidence, declared nowhere | +| `orphanAutoDeletion: true` | **removed** | inert since 1.12 | renamed `orphanResourceAutoDeletion`; the chart drops unknown keys silently | + +**Every key was proven to LAND, not assumed.** `helm template longhorn/longhorn --version 1.12.1` +(the version the chart range resolves to, per the HelmRelease's `lastAttemptedRevision`) on this +branch's own `spec.values`, substituted as `clusters/prod` substitutes them, emits all eight into the +rendered `longhorn-default-setting` ConfigMap in kebab-case. The diff against the LIVE ConfigMap is +purely additive: 8 keys added, none changed, none removed — which also confirms that removing +`orphanAutoDeletion` changes the render by nothing at all, because it was never in it. + +**The chain to the live Settings, verified read-only on prod:** all 115 `settings.longhorn.io` objects +carry `longhorn.io/configmap-resource-version: 7917` (= the ConfigMap's current `resourceVersion`) +with `status.applied: true`, and every key currently in the ConfigMap equals its Setting's value. So +the ConfigMap is authoritative and the controller re-applies on each revision. The two target +Settings sit at 5 with the key absent from the ConfigMap — they are Longhorn defaults, exactly as +measured. The final flip to 1/1 happens when this branch merges and Flux reconciles; it was +deliberately NOT hand-applied, because a `kubectl` edit is how this gap arose, and a hand-applied +ConfigMap would be silently rewritten by Helm's next upgrade. + +### The 1-vs-2 judgement, and what 1 costs + +Both went to **1**, for different reasons, and the cost is stated rather than waved at. + +* `concurrentVolumeBackupRestorePerNodeLimit: 1` **costs nothing today.** Longhorn has no backup + target configured (the `backup-target` Setting does not exist on the live cluster; it is still the + WS-STORAGE Phase-4 TODO), and velero's nightly schedule is **not** a Longhorn backup — it runs + `snapshotVolumes: false` with `defaultVolumesToFsBackup: true`, i.e. kopia file-system backup + through velero's node-agent, which this Setting does not gate. CNPG base backups are + barman/Postgres-level and are not gated either. The only consumer is an R-STATE-03 restore, where + serialising ~24 volumes is the right trade against losing etcd mid-restore. **Revisit when a + Longhorn backup target lands** — from that day it starts costing recovery time. +* `snapshotHeavyTaskConcurrentLimit: 1`, **not 2.** The generator of purge work here is rebuild + cleanup (`auto-cleanup-system-generated-snapshot` is true, there are **no** RecurringJobs, 24 + volumes hold 24 snapshots), and rebuild is already serialised to 1 per node — so purge at 1 keeps + pace with its own generator, while 2 would only drain an existing backlog faster at the price of a + second heavy writer during precisely the rebuild that starved etcd. **The cost is real and is + recorded in the runbooks:** space is reclaimed serially after a rebuild storm, the failure mode + that already produced a 37.7 GiB snapshot pinning a volume at 97.3% allocated. It is instrumented + (`LonghornDiskAllocationHigh` / `LonghornDiskUnschedulable`) and recoverable by hand; a wedged etcd + is neither. +* Headroom, so the size of the bet is on the record: I/O PSI full on the three nodes right now is + 0.30 / 0.62 / 0.37 %, and the worst measured value under a full 18-replica rebuild after node 1's + drive replacement is **3.79** against the 5% `NodeIOPressureHigh` threshold. About 1.2 points. That + is why a second concurrent heavy writer is worth refusing and why none of this is called a fix. + +### The Ansible half (`etcd_io_priority`) is NOT ported — the premise moved + +`fix/etcd-io-priority` also carried `roles/etcd_io_priority` + `playbooks/etcd-io-priority.yml`: BFQ +on the shared device, `nr_requests` 64 → 32, `IOWeight=10000` drop-ins on `system.slice` and +`k3s.service`, all read back and asserted. Assessed and declined: + +1. **The drive it was written for is gone.** §0c: node 1's 2011 C400 was replaced on 2026-09-07 and + the same load class measured I/O PSI full **11.36 → 3.79**, node Ready throughout — into node 3's + band (2.47). The branch's own §2 note concluded the shared-device premise was "right about node 2 + and wrong about node 1"; the node-1-shaped part of that premise no longer exists. +2. **It was never run against any cluster**, and its author recorded it as *unproven on this + hardware* — "it fails loudly if the settings do not land, it cannot tell you they helped" — with a + sequenced rollout (node 3 first, `--limit`, a day of `NodeIOPressureHigh` observation). That is an + operator procedure, not a merge. +3. **Its own README states it does not separate the tenants.** Both still write one ext4 filesystem + and no cgroup or scheduler control arbitrates the jbd2 journal `fdatasync` waits on. Low ceiling, + node-wide blast radius: swapping the elevator and halving `nr_requests` on the live device holding + etcd's WAL and 24 attached volumes' replicas. +4. **Its measurements are stale by construction.** Node 1 was reinstalled after the drive swap and + the founder moved to node 2; the containerd-ppid, cgroup-path and `io.latency`-absence findings all + need re-verifying on today's nodes before the asserts mean anything. + +**Recommendation:** keep it as a named operator option to revisit *if* I/O PSI climbs back toward 5% +— which, unlike in September, we would now be told about. The real fix remains §2's exit condition: +give etcd its own device. Nothing under `infra/ansible/` is touched by this branch. + +No repo-level constraint blocking `infra/ansible` commits was found (searched `ci/`, `conformance/`, +`.gitea/`, and the log): the nearest real mechanism is `ci/touched-requirements.sh`'s path map, where +`infra/*` attributes **R-ENV-01**, pulling the `dev-nodes-provisioned-as-code` L0 check — which drives +`ansible-playbook --syntax-check`/`--list-hosts` over the **dev** provisioning chain — into the gate +for any Ansible edit. Recorded because that is a real cost of touching `infra/ansible` now that dev is +decommissioned, and it is a second reason the port would not have been cheap. + +### The I/O-pressure alert was already ported — skipped, verified + +`cb28fc8`'s `NodeIOPressureHigh` / `NodeIOPressureCritical` and the dashboard I/O row are already on +`main` (as `efc6db6`), with `runbooks/node-io.md` alongside them. Verified live rather than by reading +the file: both rules are loaded in the `platform-health.nodes` group, `health: ok`, `state: inactive`, +and `ALERTS{alertname=~"NodeIOPressure.*"}` returns no samples. Nothing to port, no test added. diff --git a/platform/longhorn/helmrelease.yaml b/platform/longhorn/helmrelease.yaml index 13bc02e..da93df5 100644 --- a/platform/longhorn/helmrelease.yaml +++ b/platform/longhorn/helmrelease.yaml @@ -76,15 +76,104 @@ spec: # never blocks scheduling). replicaAutoBalance: "best-effort" - # --- rebuild I/O throttle for slow disks (etcd WAL fsync protection) --- - # The k3s VM disks sit on HDDs, so a full-replica rebuild's sequential I/O can saturate the - # spindle and starve etcd's WAL fsync — which crash-loops etcd and flaps the kube-vip VIP. - # Cap Longhorn to rebuilding ONE replica per node at a time so a node bounce / replica - # replenishment can never storm the HDDs. Deliberately conservative for this slow-disk cluster - # (Longhorn's chart default is higher). Pair with replica-replenishment-wait-interval=600 - # (Longhorn default) so brief node bounces don't trigger replenishment at all. + # --- I/O THROTTLES: bound every Longhorn write source that competes with etcd's WAL fsync --- + # WHY THIS BLOCK IS BIGGER THAN THE ONE LINE IT REPLACES. On 2026-09-07 k3s-prod-1 went + # NotReady ("Kubelet stopped posting node status") with the host up and SSH-able: its embedded + # etcd could not finish a WAL fdatasync (took=3.56s/6.71s/4.95s/3.40s against a 1 s + # expectation), "apply request took too long took=8.30s", then "etcdserver: too many + # requests". /proc/pressure/io FULL avg300 read 11.36 there against 0.75 (node 2, NVMe) and + # 1.29 (node 3). Cause: EVERY prod node puts /data/k3s (etcd WAL + containerd) and + # /data/longhorn (every replica) on ONE partition, and node 1's was a 2011 consumer SATA SSD + # with 61,440 reallocated sectors (infra/PROD-PROVISIONING.md §2 and §0c). + # + # THE HONEST HEADLINE: the rebuild path was ALREADY throttled and it is not what this block + # fixes. concurrentReplicaRebuildPerNodeLimit was 1 during that outage and it HELD — the 18 + # volumes rebuilt one at a time — and the node stalled anyway. Longhorn offers no further + # rebuild lever on the v1 data engine: replicaRebuildingBandwidthLimit is V2/SPDK ONLY + # (verified live 2026-09-13, the Setting reads a v2-only map with no v1 key) and there is no + # rebuild-concurrent-sync-limit Setting on 1.12.1 at all (NotFound). What is bounded below is + # therefore every OTHER Longhorn write source, each of which measured 5 per node. + # + # WHAT THIS BLOCK CANNOT DO. Node 1's drive HAS since been replaced (§0c), and THAT is the + # real mitigation: under the same load class — an 18-replica rebuild with node 1 as the write + # target — IO PSI full fell 11.36 -> 3.79 and the node stayed Ready. These settings only bound + # how much competing I/O Longhorn is ALLOWED to generate. etcd and Longhorn still share one + # ext4 filesystem, and no scheduler or cgroup control arbitrates the jbd2 journal that + # fdatasync waits on. 3.79 against the 5% NodeIOPressureHigh threshold is about 1.2 points of + # headroom, which is exactly why a SECOND concurrent heavy writer is still worth refusing. + # + # EVERY KEY BELOW IS PROVEN TO LAND, not assumed. `helm template longhorn/longhorn --version + # 1.12.1` (the version this chart range resolves to, per the HelmRelease's own + # lastAttemptedRevision) emits each one into the rendered longhorn-default-setting ConfigMap + # in kebab-case, and a live Setting of that name exists for each. This matters because the + # chart DROPS unknown defaultSettings keys with no error — see the orphanAutoDeletion note + # further down for the setting that was inert here for two minor versions because nobody + # checked the render. + + # Rebuild ONE replica per node at a time. Unchanged, and load-bearing: without it a node + # bounce replenishes every degraded volume at once. The freeze/thaw to "0"/"1" during an + # operator bounce (runbooks/longhorn-degraded.md, docs/plans/cluster-upgrade-plan.md) patches + # the Setting directly — deliberate and temporary; this line is the value it returns to. concurrentReplicaRebuildPerNodeLimit: 1 + # Seconds Longhorn waits before replenishing a replica on a node that went away. This is + # Longhorn's own default and the live Setting already reads 600 — PINNED, not changed, so no + # future chart or manager default can silently shorten the window in which a node can be + # recovered by hand without triggering a rebuild storm. + replicaReplenishmentWaitInterval: 600 + + # WAS 5, MEASURED LIVE ON PROD 2026-09-13. Restores read from the backup target and write + # whole volumes to the local disk — the heaviest sustained write Longhorn can produce. 5 means + # five parallel full-volume writes onto the partition etcd is fsyncing to. + # THE COST OF 1, STATED HONESTLY: today it is zero. Longhorn has NO backup target configured + # (the backup-target Setting does not even exist on the live cluster — it is still the + # WS-STORAGE Phase 4 TODO at the foot of this file), and velero's nightly run is NOT a + # Longhorn backup: its schedules set snapshotVolumes:false with defaultVolumesToFsBackup:true, + # i.e. kopia file-system backup through velero's own node-agent, which this Setting does not + # gate. CNPG base backups are barman/Postgres-level and are not gated either. The only + # consumer is an R-STATE-03 DR restore of ~24 volumes, where serialising is the correct trade + # against losing etcd mid-restore — and an operator who needs restore throughput can raise the + # Setting for the duration and watch NodeIOPressureHigh. REVISIT when a Longhorn backup target + # actually lands: from that day this knob starts costing real recovery time. + concurrentVolumeBackupRestorePerNodeLimit: 1 + + # WAS 5, MEASURED LIVE ON PROD 2026-09-13. Snapshot "heavy tasks" are purge and clone: full + # reads and rewrites of snapshot disk files. + # WHY 1 AND NOT 2 — this balance was judged, not copied. The work that generates purges here + # is rebuild cleanup: auto-cleanup-system-generated-snapshot is true, there are NO + # RecurringJobs on this cluster, and the 24 volumes hold 24 snapshots between them. Purge + # therefore arrives at the rate rebuilds retire snapshots, which is ALREADY serialised to 1 + # per node — so a purge concurrency of 1 keeps pace with its own generator. 2 would only drain + # an existing backlog faster, at the price of a second heavy writer on the disk during + # precisely the rebuild that starved etcd once, with about 1.2 points of measured PSI headroom + # to spend. + # THE COST OF 1: space is reclaimed serially after a rebuild storm. That is a REAL cost here — + # a 37.7 GiB system snapshot from prod-1's rebuild once pinned a volume at 97.3% ALLOCATED — + # but it is the cost that is already instrumented (LonghornDiskAllocationHigh, + # LonghornDiskUnschedulable) and recoverable by hand, where a wedged etcd is neither. + snapshotHeavyTaskConcurrentLimit: 1 + + # Worker threads WITHIN one backup / one restore. Both sit at Longhorn's default of 2; PINNED + # rather than lowered — combined with the per-node limit of 1 above, one volume at a time with + # two threads is already conservative, and pinning stops an upstream bump from widening it. + backupConcurrentLimit: 2 + restoreConcurrentLimit: 2 + + # Automatic engine upgrades roll every volume's engine after a Longhorn minor bump — a + # cluster-wide restart-and-resync storm on the same shared disk. 0 disables it, making engine + # upgrade a deliberate operator action, which matches this HelmRelease's own + # remediateLastFailure:false posture on minor upgrades. It is Longhorn's default; PINNED so + # both halves of that one decision live in one file. + concurrentAutomaticEngineUpgradePerNodeLimit: 0 + + # Never drain a node that holds the last healthy replica of a volume. This was ALREADY the + # live value on 2026-09-13 — but only because it is Longhorn's own default: it appeared in NO + # manifest in this repo and in NO rendered ConfigMap, so an upstream default change would have + # moved it silently. PINNED, value unchanged. runbooks/longhorn-degraded.md and + # docs/plans/cluster-upgrade-plan.md both reason about drain safety, so it has to be a + # declaration rather than a coincidence. + nodeDrainPolicy: "block-if-contains-last-replica" + # --- capacity guardrails (docs/04 "reserve disk headroom") --- # Reserve 20% of the dedicated data disk so a node never fills to 100% (which wedges the # kubelet and Longhorn alike). Dev disks are small, so headroom matters. @@ -99,8 +188,23 @@ spec: # --- misc dev-appropriate defaults --- # Keep replicas spread for durability rather than co-locating with the pod. defaultDataLocality: "disabled" - # Reclaim replica space of deleted/failed volumes promptly in dev. - orphanAutoDeletion: true + # `orphanAutoDeletion: true` USED TO BE HERE AND WAS INERT. Removed 2026-09-13, having been + # verified dead two independent ways against the DEPLOYED 1.12.1: + # 1. RENDER: `helm template longhorn/longhorn --version 1.12.1` with that key set emits NO + # orphan key whatsoever into longhorn-default-setting. Longhorn 1.9 replaced + # orphanAutoDeletion (a bool) with orphanResourceAutoDeletion (a semicolon-separated list + # such as "replica-data;instance"), and the chart template drops unknown defaultSettings + # keys with no error. + # 2. LIVE, from the other end: the rendered longhorn-default-setting ConfigMap in + # longhorn-system carries no orphan key at all, `get setting orphan-auto-deletion` is + # NotFound, and `get setting orphan-resource-auto-deletion` reads "" — Longhorn's default. + # DELIBERATELY NOT REPLACED with orphanResourceAutoDeletion: orphan deletion is extra write + # work on the one partition that is this cluster's constraint (see the I/O block above) and it + # buys only reclaimed space, of which prod has hundreds of GB free. Leaving it unset keeps + # Longhorn's default of not auto-deleting, which is the conservative side here — orphan + # cleanup stays manual, as runbooks/longhorn-capacity.md and runbooks/node-disk.md say. + # Left as a comment rather than deleted silently, because "config that looks applied and is + # not" is the failure this repo keeps re-learning, and two runbooks cite this key by name. # --- SCRAPE THE STORAGE LAYER, or every Longhorn alert is INERT (added 2026-09-13) ---------- # platform/monitoring/alerts.yaml ships the platform-health.longhorn group (as of 2026-09-13: diff --git a/platform/longhorn/values.yaml b/platform/longhorn/values.yaml index f6f318d..63e10d9 100644 --- a/platform/longhorn/values.yaml +++ b/platform/longhorn/values.yaml @@ -15,12 +15,30 @@ # replicaSoftAntiAffinity: false # HARD anti-affinity: one replica per node # replicaZoneSoftAntiAffinity: true # single zone (3 mini-PCs, no failure zones) # replicaAutoBalance: best-effort # rebalance toward one-per-node when a node returns -# concurrentReplicaRebuildPerNodeLimit: 1 # throttle rebuild I/O so it can't starve etcd WAL fsync (HDD-backed) +# +# # --- I/O throttles: keep Longhorn from starving etcd's WAL fsync. The etcd WAL and every +# # Longhorn replica share ONE partition on every prod node (PROD-PROVISIONING.md §2); see +# # helmrelease.yaml for the 2026-09-07 incident, the post-drive-replacement measurement, and the +# # cost of each value. The two marked "was 5" were measured at 5 on prod on 2026-09-13 while a +# # runbook claimed they were 1 — nothing in this repo declared them. --- +# concurrentReplicaRebuildPerNodeLimit: 1 # one replica rebuild per node at a time +# replicaReplenishmentWaitInterval: 600 # PINNED at Longhorn's default, not changed +# concurrentVolumeBackupRestorePerNodeLimit: 1 # was 5: five parallel full-volume restores +# snapshotHeavyTaskConcurrentLimit: 1 # was 5: snapshot purge/clone rewrites +# backupConcurrentLimit: 2 # PINNED at Longhorn's default (threads per backup) +# restoreConcurrentLimit: 2 # PINNED at Longhorn's default (threads per restore) +# concurrentAutomaticEngineUpgradePerNodeLimit: 0 # PINNED: engine upgrades are operator-driven +# nodeDrainPolicy: block-if-contains-last-replica # PINNED: was live only as Longhorn's default +# # NOT AVAILABLE TO US, do not add: replicaRebuildingBandwidthLimit is v2-data-engine only, and +# # no rebuild-concurrent-sync-limit Setting exists on 1.12.1 at all. +# # storageReservedPercentageForDefaultDisk: "20" # reserve disk headroom (docs/04 guardrail) # storageMinimalAvailablePercentage: "25" # stop scheduling below 25% free # storageOverProvisioningPercentage: "100" # dev: no thin over-provisioning (1:1) # defaultDataLocality: disabled # keep replicas spread for durability -# orphanAutoDeletion: true # reclaim orphaned replica space promptly +# # orphanAutoDeletion — REMOVED 2026-09-13: the key does not exist in the 1.12 chart (renamed +# # orphanResourceAutoDeletion, now a list) and was being silently dropped from the render. +# # Deliberately NOT replaced; orphan cleanup stays manual. helmrelease.yaml carries the proof. # backupTarget: # TODO(WS-STORAGE, Phase 4): DR target (ADR-0003) # # metrics: diff --git a/runbooks/longhorn-capacity.md b/runbooks/longhorn-capacity.md index 698779b..62ea29e 100644 --- a/runbooks/longhorn-capacity.md +++ b/runbooks/longhorn-capacity.md @@ -163,11 +163,17 @@ disk-full, ending in a postmaster PANIC. Go to `/runbooks/cnpg-archiving`. **Nev hand.** ### 2. Snapshots and orphaned replicas accumulating -Longhorn snapshots are not free, and cleanup here is **manual**: `orphanAutoDeletion: true` in -`platform/longhorn/helmrelease.yaml` has been **silently inert since Longhorn 1.12** — it was renamed -`orphanResourceAutoDeletion` (now a list) and the chart drops unknown `defaultSettings` keys with no -error. Verify the key's current state in that file before citing it; then prune orphans and stale -snapshots from the Longhorn UI. +Longhorn snapshots are not free, and cleanup here is **manual**. `orphanAutoDeletion: true` used to +sit in `platform/longhorn/helmrelease.yaml` and had been **silently inert since Longhorn 1.12** — it +was renamed `orphanResourceAutoDeletion` (now a list) and the chart drops unknown `defaultSettings` +keys with no error. It was **removed on 2026-09-13** and deliberately not replaced, so orphan cleanup +is manual by decision, not by accident: prune orphans and stale snapshots from the Longhorn UI. + +Note also that **snapshot purge and clone now run one at a time per node** +(`snapshotHeavyTaskConcurrentLimit: 1`, declared 2026-09-13 to keep Longhorn from starving etcd's WAL +fsync). Reclaim after a rebuild storm is therefore *serial and slower by design*. If allocation is +climbing and purge is the bottleneck, the operator may raise the Setting for the duration and watch +`NodeIOPressureHigh` — see `runbooks/longhorn-degraded.md` for the tradeoff as recorded. ### 3. A workload genuinely grew Plex/Sonarr/Radarr **media** does not live on Longhorn — it is static NFS from @@ -199,5 +205,6 @@ answer (a ~51 GB hot tier: Prometheus, CNPG, Loki, Gitea). Operator planning wor --- Source of truth: `runbooks/longhorn-capacity.md`. Index: `/runbooks/` -Records: `platform/longhorn/helmrelease.yaml` (the settings above and the inert `orphanAutoDeletion`); +Records: `platform/longhorn/helmrelease.yaml` (the settings above, the I/O throttle block, and the +removal of the inert `orphanAutoDeletion`); `platform/cnpg/alerts.yaml` header (the WAL disk-full clock); `docs/BULK-NFS-STORAGE.md`. diff --git a/runbooks/longhorn-degraded.md b/runbooks/longhorn-degraded.md index 688d7ec..c74143c 100644 --- a/runbooks/longhorn-degraded.md +++ b/runbooks/longhorn-degraded.md @@ -133,25 +133,58 @@ Thaw to `"1"` — never higher — once every volume reads `healthy`. Use `1` ra replicas are genuinely gone (an OS re-provision), so replenishment can actually proceed. `concurrentReplicaRebuildPerNodeLimit: 1` was **already in force during the 2026-09-07 outage and held**; do not "fix" this by setting a knob that is already set. `replicaRebuildingBandwidthLimit` is -**v2/SPDK-only** and does nothing here; `rebuildConcurrentSyncLimit` is already at its floor of 1. -What was *intended* after that incident was to tighten every **other** Longhorn I/O source — -`concurrentVolumeBackupRestorePerNodeLimit` and `snapshotHeavyTaskConcurrentLimit` 5 -> 1. +**v2/SPDK-only** and does nothing here, and there is no `rebuild-concurrent-sync-limit` Setting on +1.12.1 at all. What the incident actually called for was tightening every **other** Longhorn I/O +source — and as of 2026-09-13 that is declared in `platform/longhorn/helmrelease.yaml` +(`defaultSettings`), not just intended: -> **DO NOT RELY ON THAT: it was never applied.** Measured on prod 2026-09-13, -> `snapshot-heavy-task-concurrent-limit` = **5** and -> `concurrent-volume-backup-restore-per-node-limit` = **5**. Neither name appears anywhere in this -> repository, so nothing declares them and nothing would restore them. The work lives on the -> UNMERGED branch `fix/etcd-io-priority` ("cap every OTHER I/O source, and delete one setting that -> was inert"), whose tree is far behind `main` and must be cherry-picked, not merged. +| setting | declared | was, before | +|---|---|---| +| `concurrent-replica-rebuild-per-node-limit` | 1 | 1 (already declared) | +| `concurrent-volume-backup-restore-per-node-limit` | **1** | 5 (Longhorn default, undeclared) | +| `snapshot-heavy-task-concurrent-limit` | **1** | 5 (Longhorn default, undeclared) | +| `replica-replenishment-wait-interval` | 600 | 600 (pinned, not changed) | +| `backup-concurrent-limit` / `restore-concurrent-limit` | 2 / 2 | 2 / 2 (pinned) | +| `concurrent-automatic-engine-upgrade-per-node-limit` | 0 | 0 (pinned) | +| `node-drain-policy` | `block-if-contains-last-replica` | same, but as Longhorn's default only | + +> **WHAT CHANGED, AND WHAT TO CHECK BEFORE TRUSTING IT.** Until 2026-09-13 this page asserted a cap +> that did not exist: `snapshot-heavy-task-concurrent-limit` and +> `concurrent-volume-backup-restore-per-node-limit` both measured **5** on prod, neither name +> appeared anywhere in the repository, and the work sat on an unmerged branch. They are now declared +> in the HelmRelease and reconciled by Flux, which is what makes them survive a rebuild — a +> `kubectl patch` is how the gap arose in the first place. > -> Only two of the four mitigations this page implies are real: -> `concurrent-replica-rebuild-per-node-limit` = **1** (its floor, as stated) and -> `node-drain-policy` = **block-if-contains-last-replica**. Verify any limit before you trust it: -> `kubectl -n longhorn-system get setting -o jsonpath='{.value}'`. +> **Verify, do not assume** (the whole point of this note): +> `kubectl -n longhorn-system get setting -o jsonpath='{.value}'`. If a value reads 5, the +> HelmRelease has not reconciled — check `kubectl -n longhorn-system get helmrelease longhorn` and +> that `longhorn-default-setting` carries the key +> (`kubectl -n longhorn-system get cm longhorn-default-setting -o jsonpath='{.data}'`). The chart +> **drops unknown `defaultSettings` keys with no error**, so "the key is in git" is not the same as +> "the key is in effect" — `orphanAutoDeletion` was inert here for two minor versions for exactly +> that reason, and has now been removed. > -> This matters here specifically because the incident being described is etcd I/O starvation on a -> node whose WAL shares a device with Longhorn. If you are reading this page during that failure, -> the I/O cap you are counting on is not in place. +> **These caps are a bound, not a fix.** They limit how much competing I/O Longhorn is *allowed* to +> generate. etcd's WAL and every replica still share one partition on every prod node, and no +> scheduler or cgroup control arbitrates the ext4 journal that `fdatasync` waits on. The measurable +> improvement came from hardware: node 1's 2011 SATA SSD was replaced, and under the same load class +> (an 18-replica rebuild with node 1 as the write target) I/O PSI full fell **11.36 -> 3.79** and the +> node stayed Ready. 3.79 against the 5% `NodeIOPressureHigh` threshold is the headroom you have. +> +> **Cost of the two caps, so nobody is surprised during an incident:** snapshot purge and clone run +> one at a time per node, so space is reclaimed serially after a rebuild storm (watch +> `LonghornDiskAllocationHigh`); and a Longhorn *volume* restore runs one at a time per node. The +> latter costs nothing today — no Longhorn backup target is configured, and velero's nightly run is +> kopia file-system backup (`defaultVolumesToFsBackup: true`, `snapshotVolumes: false`), which this +> setting does not gate. During a real DR restore an operator may raise +> `concurrent-volume-backup-restore-per-node-limit` for the duration and watch +> `NodeIOPressureHigh` — that is a deliberate, temporary patch, like the rebuild freeze above. +> +> **Not ported, deliberately:** the same unmerged branch also carried node-level I/O arbitration +> (BFQ on the shared device, `nr_requests` 64 -> 32, systemd `IOWeight` drop-ins; +> `infra/ansible/roles/etcd_io_priority`). It was never run against any cluster, its own author +> recorded it as unproven and node-3-first, and the drive it was written for has since been replaced. +> It remains an operator option if I/O PSI climbs back toward 5%, not a merged default. ### 2. A node is still down Then the rebuild cannot complete and this will not clear. Go to `/runbooks/node-down` first; there is @@ -178,10 +211,12 @@ settling, that is worth an operator's attention. and then get the operator.** That combination is the 2026-09-07 failure mode in progress. - **Never reboot, cordon or re-provision a second node while any volume is degraded.** This is the single most important rule in this runbook. -- **Do not `kubectl drain`.** This cluster deliberately does not drain: there is no Longhorn - `nodeDrainPolicy` configured anywhere in the repo, the upgrade plan rejects - system-upgrade-controller because "its auto-drain fights this cluster's slow-HDD Longhorn", and the - node-loss conformance test evicts only its own writer pod. +- **Do not `kubectl drain`.** This cluster deliberately does not drain. `nodeDrainPolicy: + block-if-contains-last-replica` is now declared (2026-09-13; before that it was in force only as + Longhorn's own default, declared nowhere), so Longhorn will BLOCK a drain that would remove a + volume's last healthy replica — that is a backstop against the worst case, not permission to drain. + The upgrade plan still rejects system-upgrade-controller because "its auto-drain fights this + cluster's slow-HDD Longhorn", and the node-loss conformance test evicts only its own writer pod. - **Do not delete a degraded volume's replicas to "force a clean rebuild"** unless the operator agrees — with 3 replicas and one already missing you are one mistake from `/runbooks/longhorn-faulted`. - Escalate for: a node that will not come back, a disk replacement, or a volume degraded for more than diff --git a/runbooks/node-disk.md b/runbooks/node-disk.md index 9c4b06c..a9392fc 100644 --- a/runbooks/node-disk.md +++ b/runbooks/node-disk.md @@ -87,11 +87,12 @@ ssh k3s-prod-1 'sudo journalctl --disk-usage; sudo crictl images | wc -l; sudo d Hard node anti-affinity (`replicaSoftAntiAffinity: false`) means one replica per node per volume, but `replicaAutoBalance: best-effort` moves them. Prod sits around 269 GiB scheduled against an 829 GiB ceiling, so this is rarely tight — but a rebuild storm plus snapshots can spike it. -Remediation: check for orphaned replicas and stale snapshots in the Longhorn UI. Note -`orphanAutoDeletion: true` in `platform/longhorn/helmrelease.yaml` **has been silently inert since -Longhorn 1.12** — it was renamed `orphanResourceAutoDeletion` (a list) and the chart drops unknown -`defaultSettings` keys with no error. So orphan cleanup is manual. Verify the key's current state in -the file before you trust it. +Remediation: check for orphaned replicas and stale snapshots in the Longhorn UI. Orphan cleanup is +**manual**: `orphanAutoDeletion: true` sat in `platform/longhorn/helmrelease.yaml` and had been +**silently inert since Longhorn 1.12** — renamed `orphanResourceAutoDeletion` (a list), and the chart +drops unknown `defaultSettings` keys with no error. Removed 2026-09-13 and deliberately not replaced. +Snapshot purge also runs one at a time per node now (`snapshotHeavyTaskConcurrentLimit: 1`, the etcd +WAL protection), so reclaim after a rebuild is serial by design. ### 2. A CNPG database is not recycling WAL The single most dangerous cause, because it is unbounded. With `archive_mode=on` and a failing diff --git a/runbooks/node-io.md b/runbooks/node-io.md index 6a85e04..2e623ff 100644 --- a/runbooks/node-io.md +++ b/runbooks/node-io.md @@ -122,7 +122,12 @@ infra/devcontainer/dev.sh run bash -c \ Thaw to `"1"` — never higher — once all volumes read `healthy`. Use `0` when replicas will **reattach** (an in-place bounce); use `1` when they are genuinely gone (an OS re-provision). `concurrentReplicaRebuildPerNodeLimit: 1` was **already in force during the 2026-09-07 outage and -held** — do not "fix" this by setting a knob that is already set. +held** — do not "fix" this by setting a knob that is already set. Longhorn's *other* write sources +are capped as of 2026-09-13 (`snapshot-heavy-task-concurrent-limit` and +`concurrent-volume-backup-restore-per-node-limit`, both 5 -> 1, declared in +`platform/longhorn/helmrelease.yaml`), so purge/clone and volume restore can no longer run five wide +per node while a rebuild is in progress. Confirm rather than assume: +`kubectl -n longhorn-system get setting -o jsonpath='{.value}'`. ### 2. The node's disk is failing `k3s-prod-1`'s original 2011 `C400-MTFDDAC512MAM` had **61,440 reallocated sectors**, 31% endurance