diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index b330e6b..d3b44b7 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -3474,3 +3474,109 @@ volume on the cluster. after the 2026-09-07 incident. Read live on prod today: `snapshot-heavy-task-concurrent-limit` is **5**. Either the change never landed or it was reverted; the runbook is currently telling an operator a knob is set that is not. Out of scope for this branch. + +## 2026-09-13 — the Longhorn I/O caps a runbook promised are now DECLARED; the Ansible half is not ported (branch `fix/longhorn-io-caps-actually-applied`) + +Closes the "Also noticed, NOT changed" note directly above. `runbooks/longhorn-degraded.md` told +operators that `concurrentVolumeBackupRestorePerNodeLimit` and `snapshotHeavyTaskConcurrentLimit` had +been tightened 5 → 1 after the 2026-09-07 etcd-starvation incident. Measured live on prod, both read +**5**, and neither name appeared anywhere in the repository — so nothing declared them and nothing +would have restored them. This is the **fourth** instance of config that looked applied and was not +(after `orphanAutoDeletion`, the missing Longhorn ServiceMonitor, and the inert app SLI rules). + +### What landed, declaratively + +`platform/longhorn/helmrelease.yaml` `defaultSettings` — the only path Flux reconciles for these, and +`platform/longhorn/values.yaml` is the reference mirror: + +| setting | now | was | why | +|---|---|---|---| +| `concurrentVolumeBackupRestorePerNodeLimit` | **1** | 5 (undeclared default) | five parallel full-volume writes onto the partition etcd fsyncs to | +| `snapshotHeavyTaskConcurrentLimit` | **1** | 5 (undeclared default) | purge/clone rewrite whole snapshot disk files | +| `replicaReplenishmentWaitInterval` | 600 | 600 | PINNED, not changed | +| `backupConcurrentLimit` / `restoreConcurrentLimit` | 2 / 2 | 2 / 2 | PINNED, not changed | +| `concurrentAutomaticEngineUpgradePerNodeLimit` | 0 | 0 | PINNED; engine upgrade stays operator-driven | +| `nodeDrainPolicy` | `block-if-contains-last-replica` | same, **but only as Longhorn's own default** | it was in force by coincidence, declared nowhere | +| `orphanAutoDeletion: true` | **removed** | inert since 1.12 | renamed `orphanResourceAutoDeletion`; the chart drops unknown keys silently | + +**Every key was proven to LAND, not assumed.** `helm template longhorn/longhorn --version 1.12.1` +(the version the chart range resolves to, per the HelmRelease's `lastAttemptedRevision`) on this +branch's own `spec.values`, substituted as `clusters/prod` substitutes them, emits all eight into the +rendered `longhorn-default-setting` ConfigMap in kebab-case. The diff against the LIVE ConfigMap is +purely additive: 8 keys added, none changed, none removed — which also confirms that removing +`orphanAutoDeletion` changes the render by nothing at all, because it was never in it. + +**The chain to the live Settings, verified read-only on prod:** all 115 `settings.longhorn.io` objects +carry `longhorn.io/configmap-resource-version: 7917` (= the ConfigMap's current `resourceVersion`) +with `status.applied: true`, and every key currently in the ConfigMap equals its Setting's value. So +the ConfigMap is authoritative and the controller re-applies on each revision. The two target +Settings sit at 5 with the key absent from the ConfigMap — they are Longhorn defaults, exactly as +measured. The final flip to 1/1 happens when this branch merges and Flux reconciles; it was +deliberately NOT hand-applied, because a `kubectl` edit is how this gap arose, and a hand-applied +ConfigMap would be silently rewritten by Helm's next upgrade. + +### The 1-vs-2 judgement, and what 1 costs + +Both went to **1**, for different reasons, and the cost is stated rather than waved at. + +* `concurrentVolumeBackupRestorePerNodeLimit: 1` **costs nothing today.** Longhorn has no backup + target configured (the `backup-target` Setting does not exist on the live cluster; it is still the + WS-STORAGE Phase-4 TODO), and velero's nightly schedule is **not** a Longhorn backup — it runs + `snapshotVolumes: false` with `defaultVolumesToFsBackup: true`, i.e. kopia file-system backup + through velero's node-agent, which this Setting does not gate. CNPG base backups are + barman/Postgres-level and are not gated either. The only consumer is an R-STATE-03 restore, where + serialising ~24 volumes is the right trade against losing etcd mid-restore. **Revisit when a + Longhorn backup target lands** — from that day it starts costing recovery time. +* `snapshotHeavyTaskConcurrentLimit: 1`, **not 2.** The generator of purge work here is rebuild + cleanup (`auto-cleanup-system-generated-snapshot` is true, there are **no** RecurringJobs, 24 + volumes hold 24 snapshots), and rebuild is already serialised to 1 per node — so purge at 1 keeps + pace with its own generator, while 2 would only drain an existing backlog faster at the price of a + second heavy writer during precisely the rebuild that starved etcd. **The cost is real and is + recorded in the runbooks:** space is reclaimed serially after a rebuild storm, the failure mode + that already produced a 37.7 GiB snapshot pinning a volume at 97.3% allocated. It is instrumented + (`LonghornDiskAllocationHigh` / `LonghornDiskUnschedulable`) and recoverable by hand; a wedged etcd + is neither. +* Headroom, so the size of the bet is on the record: I/O PSI full on the three nodes right now is + 0.30 / 0.62 / 0.37 %, and the worst measured value under a full 18-replica rebuild after node 1's + drive replacement is **3.79** against the 5% `NodeIOPressureHigh` threshold. About 1.2 points. That + is why a second concurrent heavy writer is worth refusing and why none of this is called a fix. + +### The Ansible half (`etcd_io_priority`) is NOT ported — the premise moved + +`fix/etcd-io-priority` also carried `roles/etcd_io_priority` + `playbooks/etcd-io-priority.yml`: BFQ +on the shared device, `nr_requests` 64 → 32, `IOWeight=10000` drop-ins on `system.slice` and +`k3s.service`, all read back and asserted. Assessed and declined: + +1. **The drive it was written for is gone.** §0c: node 1's 2011 C400 was replaced on 2026-09-07 and + the same load class measured I/O PSI full **11.36 → 3.79**, node Ready throughout — into node 3's + band (2.47). The branch's own §2 note concluded the shared-device premise was "right about node 2 + and wrong about node 1"; the node-1-shaped part of that premise no longer exists. +2. **It was never run against any cluster**, and its author recorded it as *unproven on this + hardware* — "it fails loudly if the settings do not land, it cannot tell you they helped" — with a + sequenced rollout (node 3 first, `--limit`, a day of `NodeIOPressureHigh` observation). That is an + operator procedure, not a merge. +3. **Its own README states it does not separate the tenants.** Both still write one ext4 filesystem + and no cgroup or scheduler control arbitrates the jbd2 journal `fdatasync` waits on. Low ceiling, + node-wide blast radius: swapping the elevator and halving `nr_requests` on the live device holding + etcd's WAL and 24 attached volumes' replicas. +4. **Its measurements are stale by construction.** Node 1 was reinstalled after the drive swap and + the founder moved to node 2; the containerd-ppid, cgroup-path and `io.latency`-absence findings all + need re-verifying on today's nodes before the asserts mean anything. + +**Recommendation:** keep it as a named operator option to revisit *if* I/O PSI climbs back toward 5% +— which, unlike in September, we would now be told about. The real fix remains §2's exit condition: +give etcd its own device. Nothing under `infra/ansible/` is touched by this branch. + +No repo-level constraint blocking `infra/ansible` commits was found (searched `ci/`, `conformance/`, +`.gitea/`, and the log): the nearest real mechanism is `ci/touched-requirements.sh`'s path map, where +`infra/*` attributes **R-ENV-01**, pulling the `dev-nodes-provisioned-as-code` L0 check — which drives +`ansible-playbook --syntax-check`/`--list-hosts` over the **dev** provisioning chain — into the gate +for any Ansible edit. Recorded because that is a real cost of touching `infra/ansible` now that dev is +decommissioned, and it is a second reason the port would not have been cheap. + +### The I/O-pressure alert was already ported — skipped, verified + +`cb28fc8`'s `NodeIOPressureHigh` / `NodeIOPressureCritical` and the dashboard I/O row are already on +`main` (as `efc6db6`), with `runbooks/node-io.md` alongside them. Verified live rather than by reading +the file: both rules are loaded in the `platform-health.nodes` group, `health: ok`, `state: inactive`, +and `ALERTS{alertname=~"NodeIOPressure.*"}` returns no samples. Nothing to port, no test added. diff --git a/platform/longhorn/helmrelease.yaml b/platform/longhorn/helmrelease.yaml index 13bc02e..da93df5 100644 --- a/platform/longhorn/helmrelease.yaml +++ b/platform/longhorn/helmrelease.yaml @@ -76,15 +76,104 @@ spec: # never blocks scheduling). replicaAutoBalance: "best-effort" - # --- rebuild I/O throttle for slow disks (etcd WAL fsync protection) --- - # The k3s VM disks sit on HDDs, so a full-replica rebuild's sequential I/O can saturate the - # spindle and starve etcd's WAL fsync — which crash-loops etcd and flaps the kube-vip VIP. - # Cap Longhorn to rebuilding ONE replica per node at a time so a node bounce / replica - # replenishment can never storm the HDDs. Deliberately conservative for this slow-disk cluster - # (Longhorn's chart default is higher). Pair with replica-replenishment-wait-interval=600 - # (Longhorn default) so brief node bounces don't trigger replenishment at all. + # --- I/O THROTTLES: bound every Longhorn write source that competes with etcd's WAL fsync --- + # WHY THIS BLOCK IS BIGGER THAN THE ONE LINE IT REPLACES. On 2026-09-07 k3s-prod-1 went + # NotReady ("Kubelet stopped posting node status") with the host up and SSH-able: its embedded + # etcd could not finish a WAL fdatasync (took=3.56s/6.71s/4.95s/3.40s against a 1 s + # expectation), "apply request took too long took=8.30s", then "etcdserver: too many + # requests". /proc/pressure/io FULL avg300 read 11.36 there against 0.75 (node 2, NVMe) and + # 1.29 (node 3). Cause: EVERY prod node puts /data/k3s (etcd WAL + containerd) and + # /data/longhorn (every replica) on ONE partition, and node 1's was a 2011 consumer SATA SSD + # with 61,440 reallocated sectors (infra/PROD-PROVISIONING.md §2 and §0c). + # + # THE HONEST HEADLINE: the rebuild path was ALREADY throttled and it is not what this block + # fixes. concurrentReplicaRebuildPerNodeLimit was 1 during that outage and it HELD — the 18 + # volumes rebuilt one at a time — and the node stalled anyway. Longhorn offers no further + # rebuild lever on the v1 data engine: replicaRebuildingBandwidthLimit is V2/SPDK ONLY + # (verified live 2026-09-13, the Setting reads a v2-only map with no v1 key) and there is no + # rebuild-concurrent-sync-limit Setting on 1.12.1 at all (NotFound). What is bounded below is + # therefore every OTHER Longhorn write source, each of which measured 5 per node. + # + # WHAT THIS BLOCK CANNOT DO. Node 1's drive HAS since been replaced (§0c), and THAT is the + # real mitigation: under the same load class — an 18-replica rebuild with node 1 as the write + # target — IO PSI full fell 11.36 -> 3.79 and the node stayed Ready. These settings only bound + # how much competing I/O Longhorn is ALLOWED to generate. etcd and Longhorn still share one + # ext4 filesystem, and no scheduler or cgroup control arbitrates the jbd2 journal that + # fdatasync waits on. 3.79 against the 5% NodeIOPressureHigh threshold is about 1.2 points of + # headroom, which is exactly why a SECOND concurrent heavy writer is still worth refusing. + # + # EVERY KEY BELOW IS PROVEN TO LAND, not assumed. `helm template longhorn/longhorn --version + # 1.12.1` (the version this chart range resolves to, per the HelmRelease's own + # lastAttemptedRevision) emits each one into the rendered longhorn-default-setting ConfigMap + # in kebab-case, and a live Setting of that name exists for each. This matters because the + # chart DROPS unknown defaultSettings keys with no error — see the orphanAutoDeletion note + # further down for the setting that was inert here for two minor versions because nobody + # checked the render. + + # Rebuild ONE replica per node at a time. Unchanged, and load-bearing: without it a node + # bounce replenishes every degraded volume at once. The freeze/thaw to "0"/"1" during an + # operator bounce (runbooks/longhorn-degraded.md, docs/plans/cluster-upgrade-plan.md) patches + # the Setting directly — deliberate and temporary; this line is the value it returns to. concurrentReplicaRebuildPerNodeLimit: 1 + # Seconds Longhorn waits before replenishing a replica on a node that went away. This is + # Longhorn's own default and the live Setting already reads 600 — PINNED, not changed, so no + # future chart or manager default can silently shorten the window in which a node can be + # recovered by hand without triggering a rebuild storm. + replicaReplenishmentWaitInterval: 600 + + # WAS 5, MEASURED LIVE ON PROD 2026-09-13. Restores read from the backup target and write + # whole volumes to the local disk — the heaviest sustained write Longhorn can produce. 5 means + # five parallel full-volume writes onto the partition etcd is fsyncing to. + # THE COST OF 1, STATED HONESTLY: today it is zero. Longhorn has NO backup target configured + # (the backup-target Setting does not even exist on the live cluster — it is still the + # WS-STORAGE Phase 4 TODO at the foot of this file), and velero's nightly run is NOT a + # Longhorn backup: its schedules set snapshotVolumes:false with defaultVolumesToFsBackup:true, + # i.e. kopia file-system backup through velero's own node-agent, which this Setting does not + # gate. CNPG base backups are barman/Postgres-level and are not gated either. The only + # consumer is an R-STATE-03 DR restore of ~24 volumes, where serialising is the correct trade + # against losing etcd mid-restore — and an operator who needs restore throughput can raise the + # Setting for the duration and watch NodeIOPressureHigh. REVISIT when a Longhorn backup target + # actually lands: from that day this knob starts costing real recovery time. + concurrentVolumeBackupRestorePerNodeLimit: 1 + + # WAS 5, MEASURED LIVE ON PROD 2026-09-13. Snapshot "heavy tasks" are purge and clone: full + # reads and rewrites of snapshot disk files. + # WHY 1 AND NOT 2 — this balance was judged, not copied. The work that generates purges here + # is rebuild cleanup: auto-cleanup-system-generated-snapshot is true, there are NO + # RecurringJobs on this cluster, and the 24 volumes hold 24 snapshots between them. Purge + # therefore arrives at the rate rebuilds retire snapshots, which is ALREADY serialised to 1 + # per node — so a purge concurrency of 1 keeps pace with its own generator. 2 would only drain + # an existing backlog faster, at the price of a second heavy writer on the disk during + # precisely the rebuild that starved etcd once, with about 1.2 points of measured PSI headroom + # to spend. + # THE COST OF 1: space is reclaimed serially after a rebuild storm. That is a REAL cost here — + # a 37.7 GiB system snapshot from prod-1's rebuild once pinned a volume at 97.3% ALLOCATED — + # but it is the cost that is already instrumented (LonghornDiskAllocationHigh, + # LonghornDiskUnschedulable) and recoverable by hand, where a wedged etcd is neither. + snapshotHeavyTaskConcurrentLimit: 1 + + # Worker threads WITHIN one backup / one restore. Both sit at Longhorn's default of 2; PINNED + # rather than lowered — combined with the per-node limit of 1 above, one volume at a time with + # two threads is already conservative, and pinning stops an upstream bump from widening it. + backupConcurrentLimit: 2 + restoreConcurrentLimit: 2 + + # Automatic engine upgrades roll every volume's engine after a Longhorn minor bump — a + # cluster-wide restart-and-resync storm on the same shared disk. 0 disables it, making engine + # upgrade a deliberate operator action, which matches this HelmRelease's own + # remediateLastFailure:false posture on minor upgrades. It is Longhorn's default; PINNED so + # both halves of that one decision live in one file. + concurrentAutomaticEngineUpgradePerNodeLimit: 0 + + # Never drain a node that holds the last healthy replica of a volume. This was ALREADY the + # live value on 2026-09-13 — but only because it is Longhorn's own default: it appeared in NO + # manifest in this repo and in NO rendered ConfigMap, so an upstream default change would have + # moved it silently. PINNED, value unchanged. runbooks/longhorn-degraded.md and + # docs/plans/cluster-upgrade-plan.md both reason about drain safety, so it has to be a + # declaration rather than a coincidence. + nodeDrainPolicy: "block-if-contains-last-replica" + # --- capacity guardrails (docs/04 "reserve disk headroom") --- # Reserve 20% of the dedicated data disk so a node never fills to 100% (which wedges the # kubelet and Longhorn alike). Dev disks are small, so headroom matters. @@ -99,8 +188,23 @@ spec: # --- misc dev-appropriate defaults --- # Keep replicas spread for durability rather than co-locating with the pod. defaultDataLocality: "disabled" - # Reclaim replica space of deleted/failed volumes promptly in dev. - orphanAutoDeletion: true + # `orphanAutoDeletion: true` USED TO BE HERE AND WAS INERT. Removed 2026-09-13, having been + # verified dead two independent ways against the DEPLOYED 1.12.1: + # 1. RENDER: `helm template longhorn/longhorn --version 1.12.1` with that key set emits NO + # orphan key whatsoever into longhorn-default-setting. Longhorn 1.9 replaced + # orphanAutoDeletion (a bool) with orphanResourceAutoDeletion (a semicolon-separated list + # such as "replica-data;instance"), and the chart template drops unknown defaultSettings + # keys with no error. + # 2. LIVE, from the other end: the rendered longhorn-default-setting ConfigMap in + # longhorn-system carries no orphan key at all, `get setting orphan-auto-deletion` is + # NotFound, and `get setting orphan-resource-auto-deletion` reads "" — Longhorn's default. + # DELIBERATELY NOT REPLACED with orphanResourceAutoDeletion: orphan deletion is extra write + # work on the one partition that is this cluster's constraint (see the I/O block above) and it + # buys only reclaimed space, of which prod has hundreds of GB free. Leaving it unset keeps + # Longhorn's default of not auto-deleting, which is the conservative side here — orphan + # cleanup stays manual, as runbooks/longhorn-capacity.md and runbooks/node-disk.md say. + # Left as a comment rather than deleted silently, because "config that looks applied and is + # not" is the failure this repo keeps re-learning, and two runbooks cite this key by name. # --- SCRAPE THE STORAGE LAYER, or every Longhorn alert is INERT (added 2026-09-13) ---------- # platform/monitoring/alerts.yaml ships the platform-health.longhorn group (as of 2026-09-13: diff --git a/platform/longhorn/values.yaml b/platform/longhorn/values.yaml index f6f318d..63e10d9 100644 --- a/platform/longhorn/values.yaml +++ b/platform/longhorn/values.yaml @@ -15,12 +15,30 @@ # replicaSoftAntiAffinity: false # HARD anti-affinity: one replica per node # replicaZoneSoftAntiAffinity: true # single zone (3 mini-PCs, no failure zones) # replicaAutoBalance: best-effort # rebalance toward one-per-node when a node returns -# concurrentReplicaRebuildPerNodeLimit: 1 # throttle rebuild I/O so it can't starve etcd WAL fsync (HDD-backed) +# +# # --- I/O throttles: keep Longhorn from starving etcd's WAL fsync. The etcd WAL and every +# # Longhorn replica share ONE partition on every prod node (PROD-PROVISIONING.md §2); see +# # helmrelease.yaml for the 2026-09-07 incident, the post-drive-replacement measurement, and the +# # cost of each value. The two marked "was 5" were measured at 5 on prod on 2026-09-13 while a +# # runbook claimed they were 1 — nothing in this repo declared them. --- +# concurrentReplicaRebuildPerNodeLimit: 1 # one replica rebuild per node at a time +# replicaReplenishmentWaitInterval: 600 # PINNED at Longhorn's default, not changed +# concurrentVolumeBackupRestorePerNodeLimit: 1 # was 5: five parallel full-volume restores +# snapshotHeavyTaskConcurrentLimit: 1 # was 5: snapshot purge/clone rewrites +# backupConcurrentLimit: 2 # PINNED at Longhorn's default (threads per backup) +# restoreConcurrentLimit: 2 # PINNED at Longhorn's default (threads per restore) +# concurrentAutomaticEngineUpgradePerNodeLimit: 0 # PINNED: engine upgrades are operator-driven +# nodeDrainPolicy: block-if-contains-last-replica # PINNED: was live only as Longhorn's default +# # NOT AVAILABLE TO US, do not add: replicaRebuildingBandwidthLimit is v2-data-engine only, and +# # no rebuild-concurrent-sync-limit Setting exists on 1.12.1 at all. +# # storageReservedPercentageForDefaultDisk: "20" # reserve disk headroom (docs/04 guardrail) # storageMinimalAvailablePercentage: "25" # stop scheduling below 25% free # storageOverProvisioningPercentage: "100" # dev: no thin over-provisioning (1:1) # defaultDataLocality: disabled # keep replicas spread for durability -# orphanAutoDeletion: true # reclaim orphaned replica space promptly +# # orphanAutoDeletion — REMOVED 2026-09-13: the key does not exist in the 1.12 chart (renamed +# # orphanResourceAutoDeletion, now a list) and was being silently dropped from the render. +# # Deliberately NOT replaced; orphan cleanup stays manual. helmrelease.yaml carries the proof. # backupTarget: # TODO(WS-STORAGE, Phase 4): DR target (ADR-0003) # # metrics: diff --git a/runbooks/longhorn-capacity.md b/runbooks/longhorn-capacity.md index 698779b..62ea29e 100644 --- a/runbooks/longhorn-capacity.md +++ b/runbooks/longhorn-capacity.md @@ -163,11 +163,17 @@ disk-full, ending in a postmaster PANIC. Go to `/runbooks/cnpg-archiving`. **Nev hand.** ### 2. Snapshots and orphaned replicas accumulating -Longhorn snapshots are not free, and cleanup here is **manual**: `orphanAutoDeletion: true` in -`platform/longhorn/helmrelease.yaml` has been **silently inert since Longhorn 1.12** — it was renamed -`orphanResourceAutoDeletion` (now a list) and the chart drops unknown `defaultSettings` keys with no -error. Verify the key's current state in that file before citing it; then prune orphans and stale -snapshots from the Longhorn UI. +Longhorn snapshots are not free, and cleanup here is **manual**. `orphanAutoDeletion: true` used to +sit in `platform/longhorn/helmrelease.yaml` and had been **silently inert since Longhorn 1.12** — it +was renamed `orphanResourceAutoDeletion` (now a list) and the chart drops unknown `defaultSettings` +keys with no error. It was **removed on 2026-09-13** and deliberately not replaced, so orphan cleanup +is manual by decision, not by accident: prune orphans and stale snapshots from the Longhorn UI. + +Note also that **snapshot purge and clone now run one at a time per node** +(`snapshotHeavyTaskConcurrentLimit: 1`, declared 2026-09-13 to keep Longhorn from starving etcd's WAL +fsync). Reclaim after a rebuild storm is therefore *serial and slower by design*. If allocation is +climbing and purge is the bottleneck, the operator may raise the Setting for the duration and watch +`NodeIOPressureHigh` — see `runbooks/longhorn-degraded.md` for the tradeoff as recorded. ### 3. A workload genuinely grew Plex/Sonarr/Radarr **media** does not live on Longhorn — it is static NFS from @@ -199,5 +205,6 @@ answer (a ~51 GB hot tier: Prometheus, CNPG, Loki, Gitea). Operator planning wor --- Source of truth: `runbooks/longhorn-capacity.md`. Index: `/runbooks/` -Records: `platform/longhorn/helmrelease.yaml` (the settings above and the inert `orphanAutoDeletion`); +Records: `platform/longhorn/helmrelease.yaml` (the settings above, the I/O throttle block, and the +removal of the inert `orphanAutoDeletion`); `platform/cnpg/alerts.yaml` header (the WAL disk-full clock); `docs/BULK-NFS-STORAGE.md`. diff --git a/runbooks/longhorn-degraded.md b/runbooks/longhorn-degraded.md index 688d7ec..c74143c 100644 --- a/runbooks/longhorn-degraded.md +++ b/runbooks/longhorn-degraded.md @@ -133,25 +133,58 @@ Thaw to `"1"` — never higher — once every volume reads `healthy`. Use `1` ra replicas are genuinely gone (an OS re-provision), so replenishment can actually proceed. `concurrentReplicaRebuildPerNodeLimit: 1` was **already in force during the 2026-09-07 outage and held**; do not "fix" this by setting a knob that is already set. `replicaRebuildingBandwidthLimit` is -**v2/SPDK-only** and does nothing here; `rebuildConcurrentSyncLimit` is already at its floor of 1. -What was *intended* after that incident was to tighten every **other** Longhorn I/O source — -`concurrentVolumeBackupRestorePerNodeLimit` and `snapshotHeavyTaskConcurrentLimit` 5 -> 1. +**v2/SPDK-only** and does nothing here, and there is no `rebuild-concurrent-sync-limit` Setting on +1.12.1 at all. What the incident actually called for was tightening every **other** Longhorn I/O +source — and as of 2026-09-13 that is declared in `platform/longhorn/helmrelease.yaml` +(`defaultSettings`), not just intended: -> **DO NOT RELY ON THAT: it was never applied.** Measured on prod 2026-09-13, -> `snapshot-heavy-task-concurrent-limit` = **5** and -> `concurrent-volume-backup-restore-per-node-limit` = **5**. Neither name appears anywhere in this -> repository, so nothing declares them and nothing would restore them. The work lives on the -> UNMERGED branch `fix/etcd-io-priority` ("cap every OTHER I/O source, and delete one setting that -> was inert"), whose tree is far behind `main` and must be cherry-picked, not merged. +| setting | declared | was, before | +|---|---|---| +| `concurrent-replica-rebuild-per-node-limit` | 1 | 1 (already declared) | +| `concurrent-volume-backup-restore-per-node-limit` | **1** | 5 (Longhorn default, undeclared) | +| `snapshot-heavy-task-concurrent-limit` | **1** | 5 (Longhorn default, undeclared) | +| `replica-replenishment-wait-interval` | 600 | 600 (pinned, not changed) | +| `backup-concurrent-limit` / `restore-concurrent-limit` | 2 / 2 | 2 / 2 (pinned) | +| `concurrent-automatic-engine-upgrade-per-node-limit` | 0 | 0 (pinned) | +| `node-drain-policy` | `block-if-contains-last-replica` | same, but as Longhorn's default only | + +> **WHAT CHANGED, AND WHAT TO CHECK BEFORE TRUSTING IT.** Until 2026-09-13 this page asserted a cap +> that did not exist: `snapshot-heavy-task-concurrent-limit` and +> `concurrent-volume-backup-restore-per-node-limit` both measured **5** on prod, neither name +> appeared anywhere in the repository, and the work sat on an unmerged branch. They are now declared +> in the HelmRelease and reconciled by Flux, which is what makes them survive a rebuild — a +> `kubectl patch` is how the gap arose in the first place. > -> Only two of the four mitigations this page implies are real: -> `concurrent-replica-rebuild-per-node-limit` = **1** (its floor, as stated) and -> `node-drain-policy` = **block-if-contains-last-replica**. Verify any limit before you trust it: -> `kubectl -n longhorn-system get setting -o jsonpath='{.value}'`. +> **Verify, do not assume** (the whole point of this note): +> `kubectl -n longhorn-system get setting -o jsonpath='{.value}'`. If a value reads 5, the +> HelmRelease has not reconciled — check `kubectl -n longhorn-system get helmrelease longhorn` and +> that `longhorn-default-setting` carries the key +> (`kubectl -n longhorn-system get cm longhorn-default-setting -o jsonpath='{.data}'`). The chart +> **drops unknown `defaultSettings` keys with no error**, so "the key is in git" is not the same as +> "the key is in effect" — `orphanAutoDeletion` was inert here for two minor versions for exactly +> that reason, and has now been removed. > -> This matters here specifically because the incident being described is etcd I/O starvation on a -> node whose WAL shares a device with Longhorn. If you are reading this page during that failure, -> the I/O cap you are counting on is not in place. +> **These caps are a bound, not a fix.** They limit how much competing I/O Longhorn is *allowed* to +> generate. etcd's WAL and every replica still share one partition on every prod node, and no +> scheduler or cgroup control arbitrates the ext4 journal that `fdatasync` waits on. The measurable +> improvement came from hardware: node 1's 2011 SATA SSD was replaced, and under the same load class +> (an 18-replica rebuild with node 1 as the write target) I/O PSI full fell **11.36 -> 3.79** and the +> node stayed Ready. 3.79 against the 5% `NodeIOPressureHigh` threshold is the headroom you have. +> +> **Cost of the two caps, so nobody is surprised during an incident:** snapshot purge and clone run +> one at a time per node, so space is reclaimed serially after a rebuild storm (watch +> `LonghornDiskAllocationHigh`); and a Longhorn *volume* restore runs one at a time per node. The +> latter costs nothing today — no Longhorn backup target is configured, and velero's nightly run is +> kopia file-system backup (`defaultVolumesToFsBackup: true`, `snapshotVolumes: false`), which this +> setting does not gate. During a real DR restore an operator may raise +> `concurrent-volume-backup-restore-per-node-limit` for the duration and watch +> `NodeIOPressureHigh` — that is a deliberate, temporary patch, like the rebuild freeze above. +> +> **Not ported, deliberately:** the same unmerged branch also carried node-level I/O arbitration +> (BFQ on the shared device, `nr_requests` 64 -> 32, systemd `IOWeight` drop-ins; +> `infra/ansible/roles/etcd_io_priority`). It was never run against any cluster, its own author +> recorded it as unproven and node-3-first, and the drive it was written for has since been replaced. +> It remains an operator option if I/O PSI climbs back toward 5%, not a merged default. ### 2. A node is still down Then the rebuild cannot complete and this will not clear. Go to `/runbooks/node-down` first; there is @@ -178,10 +211,12 @@ settling, that is worth an operator's attention. and then get the operator.** That combination is the 2026-09-07 failure mode in progress. - **Never reboot, cordon or re-provision a second node while any volume is degraded.** This is the single most important rule in this runbook. -- **Do not `kubectl drain`.** This cluster deliberately does not drain: there is no Longhorn - `nodeDrainPolicy` configured anywhere in the repo, the upgrade plan rejects - system-upgrade-controller because "its auto-drain fights this cluster's slow-HDD Longhorn", and the - node-loss conformance test evicts only its own writer pod. +- **Do not `kubectl drain`.** This cluster deliberately does not drain. `nodeDrainPolicy: + block-if-contains-last-replica` is now declared (2026-09-13; before that it was in force only as + Longhorn's own default, declared nowhere), so Longhorn will BLOCK a drain that would remove a + volume's last healthy replica — that is a backstop against the worst case, not permission to drain. + The upgrade plan still rejects system-upgrade-controller because "its auto-drain fights this + cluster's slow-HDD Longhorn", and the node-loss conformance test evicts only its own writer pod. - **Do not delete a degraded volume's replicas to "force a clean rebuild"** unless the operator agrees — with 3 replicas and one already missing you are one mistake from `/runbooks/longhorn-faulted`. - Escalate for: a node that will not come back, a disk replacement, or a volume degraded for more than diff --git a/runbooks/node-disk.md b/runbooks/node-disk.md index 9c4b06c..a9392fc 100644 --- a/runbooks/node-disk.md +++ b/runbooks/node-disk.md @@ -87,11 +87,12 @@ ssh k3s-prod-1 'sudo journalctl --disk-usage; sudo crictl images | wc -l; sudo d Hard node anti-affinity (`replicaSoftAntiAffinity: false`) means one replica per node per volume, but `replicaAutoBalance: best-effort` moves them. Prod sits around 269 GiB scheduled against an 829 GiB ceiling, so this is rarely tight — but a rebuild storm plus snapshots can spike it. -Remediation: check for orphaned replicas and stale snapshots in the Longhorn UI. Note -`orphanAutoDeletion: true` in `platform/longhorn/helmrelease.yaml` **has been silently inert since -Longhorn 1.12** — it was renamed `orphanResourceAutoDeletion` (a list) and the chart drops unknown -`defaultSettings` keys with no error. So orphan cleanup is manual. Verify the key's current state in -the file before you trust it. +Remediation: check for orphaned replicas and stale snapshots in the Longhorn UI. Orphan cleanup is +**manual**: `orphanAutoDeletion: true` sat in `platform/longhorn/helmrelease.yaml` and had been +**silently inert since Longhorn 1.12** — renamed `orphanResourceAutoDeletion` (a list), and the chart +drops unknown `defaultSettings` keys with no error. Removed 2026-09-13 and deliberately not replaced. +Snapshot purge also runs one at a time per node now (`snapshotHeavyTaskConcurrentLimit: 1`, the etcd +WAL protection), so reclaim after a rebuild is serial by design. ### 2. A CNPG database is not recycling WAL The single most dangerous cause, because it is unbounded. With `archive_mode=on` and a failing diff --git a/runbooks/node-io.md b/runbooks/node-io.md index 6a85e04..2e623ff 100644 --- a/runbooks/node-io.md +++ b/runbooks/node-io.md @@ -122,7 +122,12 @@ infra/devcontainer/dev.sh run bash -c \ Thaw to `"1"` — never higher — once all volumes read `healthy`. Use `0` when replicas will **reattach** (an in-place bounce); use `1` when they are genuinely gone (an OS re-provision). `concurrentReplicaRebuildPerNodeLimit: 1` was **already in force during the 2026-09-07 outage and -held** — do not "fix" this by setting a knob that is already set. +held** — do not "fix" this by setting a knob that is already set. Longhorn's *other* write sources +are capped as of 2026-09-13 (`snapshot-heavy-task-concurrent-limit` and +`concurrent-volume-backup-restore-per-node-limit`, both 5 -> 1, declared in +`platform/longhorn/helmrelease.yaml`), so purge/clone and volume restore can no longer run five wide +per node while a rebuild is in progress. Confirm rather than assume: +`kubectl -n longhorn-system get setting -o jsonpath='{.value}'`. ### 2. The node's disk is failing `k3s-prod-1`'s original 2011 `C400-MTFDDAC512MAM` had **61,440 reallocated sectors**, 31% endurance