fix(longhorn): cap concurrent-replica-rebuild-per-node-limit=1 (HDD etcd protection)

The k3s VM disks are HDD-backed, so a full Longhorn replica rebuild's
sequential I/O can saturate the spindle and starve etcd's WAL fsync,
crash-looping etcd and flapping the kube-vip VIP (observed during a
rolling node resize). Cap rebuilds to one replica per node at a time so a
node bounce or replica replenishment can never storm the disks.

Set via HelmRelease defaultSettings.concurrentReplicaRebuildPerNodeLimit
(GitOps-managed, permanent). replica-replenishment-wait-interval stays at
the Longhorn default of 600s. Reference values.yaml kept in sync.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 4.8 committed 2026-08-29 15:03:03 -05:00
1 parent 7910128675
commit 5c64e06354
2 files changed
+10

No files matched your search

+1
View File
@@ -13,6 +13,7 @@
# replicaSoftAntiAffinity: false # HARD anti-affinity: one replica per node
# replicaZoneSoftAntiAffinity: true # single zone (3 mini-PCs, no failure zones)
# replicaAutoBalance: best-effort # rebalance toward one-per-node when a node returns
# concurrentReplicaRebuildPerNodeLimit: 1 # throttle rebuild I/O so it can't starve etcd WAL fsync (HDD-backed)
# storageReservedPercentageForDefaultDisk: "20" # reserve disk headroom (docs/04 guardrail)
# storageMinimalAvailablePercentage: "25" # stop scheduling below 25% free
# storageOverProvisioningPercentage: "100" # dev: no thin over-provisioning (1:1)