jakman85andClaude Opus 5
f663e7c801
`runbooks/longhorn-degraded.md` told operators that, after the 2026-09-07 etcd
I/O-starvation incident, `concurrentVolumeBackupRestorePerNodeLimit` and
`snapshotHeavyTaskConcurrentLimit` had been tightened 5 -> 1. Measured live on
prod today: both read **5**, and neither name appeared anywhere in this
repository — so nothing declared them and nothing would have restored them.
Fourth instance of config that looked applied and was not.
Landed in `platform/longhorn/helmrelease.yaml` `defaultSettings` (the path Flux
reconciles; `values.yaml` is the reference mirror), NOT by `kubectl patch` —
a hand-applied Setting is exactly how this gap arose and Helm would silently
rewrite it on its next upgrade.
concurrentVolumeBackupRestorePerNodeLimit 5 -> 1
snapshotHeavyTaskConcurrentLimit 5 -> 1
replicaReplenishmentWaitInterval 600 PINNED, unchanged
backupConcurrentLimit / restoreConcurrentLimit 2 / 2 PINNED, unchanged
concurrentAutomaticEngineUpgradePerNodeLimit 0 PINNED, unchanged
nodeDrainPolicy block-if-contains-last-replica PINNED — it was in
force only as Longhorn's OWN DEFAULT, declared in no manifest and in no
rendered ConfigMap, so an upstream default change would have moved it
silently while two runbooks reasoned about drain safety from it.
EVERY KEY PROVEN TO LAND, not assumed. `helm template longhorn/longhorn
--version 1.12.1` (the version this chart range resolves to, per the
HelmRelease's own lastAttemptedRevision) on this branch's spec.values,
substituted as clusters/prod substitutes them, emits all eight into the rendered
`longhorn-default-setting` ConfigMap in kebab-case. Diffed against the LIVE
ConfigMap the change is purely ADDITIVE: 8 keys added, none altered, none lost.
The chain to the Settings is verified read-only on prod — all 115
settings.longhorn.io carry `longhorn.io/configmap-resource-version: 7917`
(= that ConfigMap's resourceVersion) with `status.applied: true`, and every key
in the ConfigMap equals its Setting's value. The two targets sit at 5 with the
key absent, i.e. Longhorn defaults, as measured.
`orphanAutoDeletion: true` REMOVED, verified inert two independent ways against
the deployed 1.12.1: the render emits no orphan key at all (Longhorn 1.9 renamed
it `orphanResourceAutoDeletion`, a list, and the chart drops unknown
defaultSettings keys with no error), and live, `setting/orphan-auto-deletion` is
NotFound while `orphan-resource-auto-deletion` reads "". Deliberately NOT
replaced — orphan deletion is extra writes on the constrained partition and buys
only space. Orphan cleanup stays manual by decision; the two runbooks that cite
the key by name are corrected rather than left to go stale.
BOTH NEW CAPS ARE 1, AND 1 COSTS SOMETHING — stated, not waved at.
`concurrentVolumeBackupRestorePerNodeLimit: 1` costs nothing TODAY: Longhorn has
no backup target (the `backup-target` Setting does not exist; still the
WS-STORAGE Phase-4 TODO), and velero's nightly run is not a Longhorn backup —
`snapshotVolumes: false` + `defaultVolumesToFsBackup: true` is kopia
file-system backup through velero's node-agent, which this Setting does not
gate; CNPG base backups are barman-level and ungated too. Its only consumer is
an R-STATE-03 restore, where serialising ~24 volumes beats losing etcd
mid-restore. Revisit when a Longhorn backup target lands.
`snapshotHeavyTaskConcurrentLimit: 1` rather than 2 because purge work is
generated by rebuild cleanup (auto-cleanup-system-generated-snapshot true, NO
RecurringJobs, 24 volumes / 24 snapshots) and rebuild is already serialised to
1 per node — so 1 keeps pace with its own generator, where 2 buys backlog drain
at the price of a second heavy writer during the exact rebuild that starved
etcd. The cost: space reclaims serially after a rebuild storm — real here (a
37.7 GiB snapshot once pinned a volume at 97.3% allocated), but instrumented by
LonghornDiskAllocationHigh and recoverable by hand, where a wedged etcd is not.
Headroom on the record: I/O PSI full is 0.30/0.62/0.37 % now, and 3.79 under a
full 18-replica rebuild after node 1's drive replacement, against a 5% warning
threshold. These caps are a BOUND, not a fix — etcd and Longhorn still share one
ext4 filesystem and nothing arbitrates the jbd2 journal fdatasync waits on.
NOT PORTED, deliberately: the same unmerged branch's node-level I/O arbitration
(`roles/etcd_io_priority` — BFQ on the shared device, nr_requests 64 -> 32,
systemd IOWeight drop-ins). The 2011 C400 it was written for was replaced on
2026-09-07 and the same load class measured 11.36 -> 3.79 I/O PSI with the node
Ready throughout; the role was never run against any cluster, its author
recorded it as unproven with a node-3-first sequenced rollout, and its own
README concedes it does not separate the two tenants. Node-wide blast radius,
low ceiling, stale measurements (node 1 reinstalled, founder moved to node 2).
It stays an operator option if I/O PSI climbs back toward 5% — which, unlike in
September, we would now be told about. Nothing under infra/ansible/ is touched.
ALSO ALREADY DONE, so skipped: that branch's NodeIOPressureHigh/Critical and the
dashboard I/O row are on main already (efc6db6) with runbooks/node-io.md.
Verified live, not by reading the file: both rules loaded in
platform-health.nodes, health ok, state inactive, and
ALERTS{alertname=~"NodeIOPressure.*"} returns no samples.
Runbooks corrected in both directions: longhorn-degraded.md's prominent "DO NOT
RELY ON THAT: it was never applied" block becomes an accurate statement of what
is now declared, what to verify before trusting it, what it cannot do, and what
it costs; longhorn-capacity.md / node-disk.md stop citing a key that no longer
exists; node-io.md gains the one line an operator needs mid-incident.
No new ${...} substitution variables and not one dollar sign added anywhere in
the diff, so neither StrictPostBuildSubstitutions nor the L0 doubled-dollar guard
is affected.
Co-Authored-By: Claude Opus 5 <[email protected]>
50 lines
3.7 KiB
YAML
50 lines
3.7 KiB
YAML
# Longhorn Helm values — REFERENCE ONLY (not consumed by kustomization.yaml).
|
|
#
|
|
# The authoritative values are inline in helmrelease.yaml (matching the sibling pattern across
|
|
# platform/ — cilium, cert-manager, velero, etc.). This file documents the knobs that matter so a
|
|
# reviewer can see intent without reading the HelmRelease. Keep it in sync with helmrelease.yaml.
|
|
#
|
|
# persistence:
|
|
# defaultClass: false # we own StorageClasses (storageclasses.yaml)
|
|
#
|
|
# defaultSettings:
|
|
# defaultDataPath: ${LONGHORN_DATA_PATH} # cluster-vars knob: dev /var/lib/longhorn
|
|
# # (dedicated /dev/vdb), prod /data/longhorn
|
|
# # (a subdirectory of the shared data partition)
|
|
# defaultReplicaCount: ${LONGHORN_DEFAULT_REPLICA_COUNT} # overlay knob (=3 in dev)
|
|
# replicaSoftAntiAffinity: false # HARD anti-affinity: one replica per node
|
|
# replicaZoneSoftAntiAffinity: true # single zone (3 mini-PCs, no failure zones)
|
|
# replicaAutoBalance: best-effort # rebalance toward one-per-node when a node returns
|
|
#
|
|
# # --- I/O throttles: keep Longhorn from starving etcd's WAL fsync. The etcd WAL and every
|
|
# # Longhorn replica share ONE partition on every prod node (PROD-PROVISIONING.md §2); see
|
|
# # helmrelease.yaml for the 2026-09-07 incident, the post-drive-replacement measurement, and the
|
|
# # cost of each value. The two marked "was 5" were measured at 5 on prod on 2026-09-13 while a
|
|
# # runbook claimed they were 1 — nothing in this repo declared them. ---
|
|
# concurrentReplicaRebuildPerNodeLimit: 1 # one replica rebuild per node at a time
|
|
# replicaReplenishmentWaitInterval: 600 # PINNED at Longhorn's default, not changed
|
|
# concurrentVolumeBackupRestorePerNodeLimit: 1 # was 5: five parallel full-volume restores
|
|
# snapshotHeavyTaskConcurrentLimit: 1 # was 5: snapshot purge/clone rewrites
|
|
# backupConcurrentLimit: 2 # PINNED at Longhorn's default (threads per backup)
|
|
# restoreConcurrentLimit: 2 # PINNED at Longhorn's default (threads per restore)
|
|
# concurrentAutomaticEngineUpgradePerNodeLimit: 0 # PINNED: engine upgrades are operator-driven
|
|
# nodeDrainPolicy: block-if-contains-last-replica # PINNED: was live only as Longhorn's default
|
|
# # NOT AVAILABLE TO US, do not add: replicaRebuildingBandwidthLimit is v2-data-engine only, and
|
|
# # no rebuild-concurrent-sync-limit Setting exists on 1.12.1 at all.
|
|
#
|
|
# storageReservedPercentageForDefaultDisk: "20" # reserve disk headroom (docs/04 guardrail)
|
|
# storageMinimalAvailablePercentage: "25" # stop scheduling below 25% free
|
|
# storageOverProvisioningPercentage: "100" # dev: no thin over-provisioning (1:1)
|
|
# defaultDataLocality: disabled # keep replicas spread for durability
|
|
# # orphanAutoDeletion — REMOVED 2026-09-13: the key does not exist in the 1.12 chart (renamed
|
|
# # orphanResourceAutoDeletion, now a list) and was being silently dropped from the render.
|
|
# # Deliberately NOT replaced; orphan cleanup stays manual. helmrelease.yaml carries the proof.
|
|
# backupTarget: <off-cluster> # TODO(WS-STORAGE, Phase 4): DR target (ADR-0003)
|
|
#
|
|
# metrics:
|
|
# serviceMonitor:
|
|
# enabled: true # scrape longhorn-manager (port `manager`, 9500);
|
|
# # without it the whole longhorn_* family is absent
|
|
# # from the TSDB and the four Longhorn alert rules
|
|
# # in platform/monitoring/alerts.yaml are INERT
|