jakman85andClaude Opus 5
4687521d65
LonghornVolumeCapacityHigh went pending within minutes of the Longhorn scrape
landing today. Re-measured on prod: it read 0.9731 on the Prometheus TSDB volume
(actual_size 52243681280 / capacity 53687091200) while the kubelet reported the
SAME filesystem 20.8% used. actual_size is head + pinned snapshots, and one
user_created=false system snapshot from prod-1's replica rebuild holds 40510484480
bytes (37.7 GiB) of it; Longhorn returns no freed blocks without discard/TRIM, and
retentionSize 45GiB on a 50Gi volume targets 90% by design. The ratio is pinned
near 1.0 and cannot fall back under 0.85 — a permanently-firing alert, the same
shape as CNPGNoRecoverabilityPoint.
DELETED, not re-thresholded. Raising the threshold would tune away a true
statement about allocation. Redundancy proven, not asserted: of 24 Longhorn
volumes on prod, 23 appear in kubelet_volume_stats_inodes > 0 and so fall inside
MonitoringVolumeFillingUp (namespace="monitoring") or PersistentVolumeFillingUp /
PersistentVolumeAlmostFull. The 24th is unifi/unifi-rootdisk, volumeMode Block —
no filesystem for anyone to measure, and the deleted rule was not covering it
either.
AND A SECOND INERTNESS the scrape did not fix. longhorn_volume_robustness on
longhorn 1.12.1 is one-hot over a `state` label with values 0/1, not the 0-3 enum
the rules assumed:
count(longhorn_volume_robustness) => 96 (24 volumes x 4 states)
count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
count(longhorn_volume_robustness == 2) => (empty)
count(longhorn_volume_robustness == 3) => (empty)
So a FAULTED volume would still have alerted nobody. Re-keyed to the label; the
"44 series / 72 unknown(0, detached)" note in platform/longhorn/helmrelease.yaml
was the same misreading and is corrected in place.
The allocation question moved to where allocation is a budget. LonghornNodeStorageHigh
fired at 0.85, but storage-minimal-available-percentage is 25 on this cluster, so a
disk leaves the scheduling pool at 75% used — the old threshold could only fire
after scheduling had already been refused. Replaced by:
LonghornDiskAllocationHigh (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70, 30m
LonghornDiskUnschedulable longhorn_disk_status{condition="schedulable"} == 0, 15m
Per DISK, not per node: longhorn_node_storage_usage_bytes sums a node's disks, so
one full disk could hide behind a half-empty sibling. Measured prod usage
0.2254 / 0.2306 / 0.2369 on prod-1/2/3.
PROVEN NEITHER PERMANENTLY FIRING NOR INERT. platform/monitoring/tests/longhorn.test.yaml
adds 6 cases pinning every rule quiet on today's measured prod state and firing on
a constructed bad state, with every `for:` asserted, plus regression assertions that
the old `== 2`/`== 3` forms match nothing and that per-disk 0.72 is visible where the
node-summed 0.475 is not. Mutation-checked 11 ways; each turns the suite RED. Live on
prod: all four new expressions return empty, while the sum of their selector
cardinalities is 54. The L0 UNTESTED_ALERTS register shrinks by seven (7/56 -> 14/58
alert rules unit-tested), which also brings MonitoringVolumeFillingUp,
PersistentVolumeFillingUp and PersistentVolumeAlmostFull under test since the
redundancy claim rests on them.
REPORTED, NOT CHANGED (DECISIONS-LOG, owner decisions): retentionSize 45GiB on a
50Gi volume is too tight — recommend 36GiB, with reasoning; and discard/TRIM on the
longhorn StorageClass is feasible but not advisable now — remove-snapshots-during-
filesystem-trim reads false on prod, so trim would not have reclaimed the 37.7 GiB
anyway, and continuous discard on nodes where the etcd WAL shares the spindle points
at this cluster's known failure mode.
Co-Authored-By: Claude Opus 5 <[email protected]>
173 lines
11 KiB
YAML
173 lines
11 KiB
YAML
---
|
|
# Longhorn — distributed replicated block storage across the 3 nodes (ADR-0003).
|
|
# Serves R-STATE-01 (state survives destroy/rebuild) and R-STATE-02 (3-way replication).
|
|
#
|
|
# Dev-appropriate values. Environment differences enter ONLY via $${...} substitution from
|
|
# cluster-vars (R-ENV-02); everything else is identical base config for dev and prod.
|
|
apiVersion: helm.toolkit.fluxcd.io/v2
|
|
kind: HelmRelease
|
|
metadata:
|
|
name: longhorn
|
|
namespace: longhorn-system
|
|
spec:
|
|
interval: 30m
|
|
# Longhorn upgrades roll every workload + engine image on slow HDD-backed nodes and can exceed the
|
|
# 5m Helm default; a short timeout caused a W5 mid-upgrade wedge (data migrated to the new minor but
|
|
# workloads rolled back, and the new-version data refuses the old-version manager). Give it room.
|
|
timeout: 15m0s
|
|
chart:
|
|
spec:
|
|
chart: longhorn
|
|
version: ">=1.12.0 <1.13.0" # W5 upgrade 1.11->1.12 (resolves 1.12.1; supports k8s 1.36). Renovate maintains
|
|
sourceRef:
|
|
kind: HelmRepository
|
|
name: longhorn
|
|
namespace: longhorn-system
|
|
install:
|
|
createNamespace: false # namespace.yaml owns it (GitOps-managed, labelled)
|
|
remediation:
|
|
retries: 3
|
|
upgrade:
|
|
remediation:
|
|
retries: 3
|
|
# NEVER auto-rollback a Longhorn minor upgrade: once the on-disk data migrates to the new minor,
|
|
# the older manager fatally refuses it ("downgrading from vX to vY is not supported"). Rolling
|
|
# back the workloads while data is already migrated wedges the cluster. Fail forward instead.
|
|
remediateLastFailure: false
|
|
values:
|
|
# We own the StorageClasses (storageclasses.yaml) so the default 3-replica class and the bulk
|
|
# tier are explicit and GitOps-managed. Disable the chart's built-in default SC so there is
|
|
# exactly one default-class annotation in the cluster.
|
|
persistence:
|
|
defaultClass: false
|
|
|
|
defaultSettings:
|
|
# --- data path (R-STATE-02, R-ENV-02) ---
|
|
# The on-node directory every replica is written into. A cluster-vars KNOB, not a literal,
|
|
# because the two environments mount their data disk differently and a hardcoded path would
|
|
# put one of them on the OS filesystem:
|
|
# dev /var/lib/longhorn — a dedicated ext4 disk (/dev/vdb) mounted there by
|
|
# roles/node_baseline. The mount root IS the dataPath: one disk,
|
|
# one tenant. 18 live volumes; this value must never change.
|
|
# prod /data/longhorn — a SUBDIRECTORY of the claimed ~424 GB partition mounted at
|
|
# /data, whose sibling /data/k3s holds k3s's entire data dir
|
|
# (etcd + the containerd image store, moved off the 11.8 GiB
|
|
# /var). Two tenants, so Longhorn gets a subdirectory rather
|
|
# than the mount root. infra/PROD-PROVISIONING.md §2.
|
|
# [parity] MUST equal roles/node_baseline's node_baseline_longhorn_data_path for the same
|
|
# environment — Ansible creates and mounts the directory, Helm tells Longhorn to use it.
|
|
# Set in BOTH overlays (clusters/dev, clusters/prod); base default is the dev value.
|
|
defaultDataPath: "${LONGHORN_DATA_PATH}"
|
|
|
|
# --- replication (R-STATE-02) ---
|
|
# Global fallback replica count. The `longhorn` StorageClass sets numberOfReplicas explicitly
|
|
# (authoritative); this keeps any volume created without an explicit count consistent.
|
|
# Substituted from cluster-vars.longhorn_default_replica_count (=3 in dev).
|
|
defaultReplicaCount: ${LONGHORN_DEFAULT_REPLICA_COUNT}
|
|
|
|
# --- replica anti-affinity across nodes (R-STATE-02 durability guarantee) ---
|
|
# HARD anti-affinity: never place two replicas of the same volume on the same node, so a
|
|
# single node loss cannot take down more than one replica. On a 3-node / 3-replica cluster
|
|
# this yields exactly one replica per node.
|
|
replicaSoftAntiAffinity: false
|
|
# Single zone (3 mini-PCs, no failure zones): allow replicas within the one zone.
|
|
replicaZoneSoftAntiAffinity: true
|
|
# When a downed node returns, rebalance replicas back toward one-per-node (best-effort so it
|
|
# never blocks scheduling).
|
|
replicaAutoBalance: "best-effort"
|
|
|
|
# --- rebuild I/O throttle for slow disks (etcd WAL fsync protection) ---
|
|
# The k3s VM disks sit on HDDs, so a full-replica rebuild's sequential I/O can saturate the
|
|
# spindle and starve etcd's WAL fsync — which crash-loops etcd and flaps the kube-vip VIP.
|
|
# Cap Longhorn to rebuilding ONE replica per node at a time so a node bounce / replica
|
|
# replenishment can never storm the HDDs. Deliberately conservative for this slow-disk cluster
|
|
# (Longhorn's chart default is higher). Pair with replica-replenishment-wait-interval=600
|
|
# (Longhorn default) so brief node bounces don't trigger replenishment at all.
|
|
concurrentReplicaRebuildPerNodeLimit: 1
|
|
|
|
# --- capacity guardrails (docs/04 "reserve disk headroom") ---
|
|
# Reserve 20% of the dedicated data disk so a node never fills to 100% (which wedges the
|
|
# kubelet and Longhorn alike). Dev disks are small, so headroom matters.
|
|
storageReservedPercentageForDefaultDisk: "20"
|
|
# Stop scheduling new replicas onto a disk once free space drops below 25%.
|
|
storageMinimalAvailablePercentage: "25"
|
|
# Dev: 200% thin over-provisioning — config-only volumes use little actual space, so the small
|
|
# 39Gi data disks host the full app catalog. Physical fill risk is low (watch monitoring PVCs).
|
|
# Prod-grade alternative: grow the Longhorn data disks and return to 1:1 (WS-STORAGE).
|
|
storageOverProvisioningPercentage: "200"
|
|
|
|
# --- misc dev-appropriate defaults ---
|
|
# Keep replicas spread for durability rather than co-locating with the pod.
|
|
defaultDataLocality: "disabled"
|
|
# Reclaim replica space of deleted/failed volumes promptly in dev.
|
|
orphanAutoDeletion: true
|
|
|
|
# --- SCRAPE THE STORAGE LAYER, or every Longhorn alert is INERT (added 2026-09-13) ----------
|
|
# platform/monitoring/alerts.yaml ships the platform-health.longhorn group (as of 2026-09-13:
|
|
# LonghornVolumeDegraded, LonghornVolumeFaulted, LonghornDiskAllocationHigh and
|
|
# LonghornDiskUnschedulable) — and NONE of them could ever fire, because the chart defaults
|
|
# `metrics.serviceMonitor.enabled: false` and no ServiceMonitor existed in ns longhorn-system.
|
|
# VERIFIED live on prod 2026-09-13: `count by (job) (up)` listed 19 jobs and none was Longhorn;
|
|
# `count(longhorn_volume_robustness)` and `count({__name__=~"longhorn.*"})` both returned NOTHING.
|
|
# So the entire longhorn_* family was absent from the TSDB and four alert rules were decoration.
|
|
#
|
|
# WHY THIS IS PART OF THE STORAGE PATH, NOT AN OBSERVABILITY NICE-TO-HAVE: this cluster's
|
|
# recurring failure mode is a disk. prod-1's failing SSD starved etcd's WAL fsync and wedged the
|
|
# node (DECISIONS-LOG; the prod-1 drive replacement), and a degraded/faulted replica is precisely
|
|
# the first symptom Longhorn publishes. R-STATE-02 promises 3-way replication survives a node
|
|
# loss; with no scrape, a volume can sit at 2 replicas — or faulted — indefinitely, unalerted.
|
|
# platform/monitoring/pvc-capacity-alerts.yaml's header already flagged this exact gap as "a
|
|
# separate defect (a missing Longhorn ServiceMonitor/metrics config)". This closes it.
|
|
#
|
|
# Chart mechanics, verified by rendering longhorn 1.12.1 (the version this range resolves to):
|
|
# * templates/servicemonitor.yaml is gated on `{{- if .Values.metrics.serviceMonitor.enabled }}`
|
|
# and NOTHING else — no `.Capabilities` probe. Measured: `helm template --set
|
|
# metrics.serviceMonitor.enabled=true` emits exactly 1 ServiceMonitor, and 0 without the flag.
|
|
# So this line is both necessary and sufficient; unlike velero's, it needs no API capability.
|
|
# * the rendered ServiceMonitor selects `app: longhorn-manager` in the release namespace on the
|
|
# port named `manager`. CONFIRMED against prod: Service/longhorn-backend carries
|
|
# `app: longhorn-manager` and a port `manager` (9500) — the selector resolves to real
|
|
# endpoints, it is not a dangling monitor (which is how the apps/ SLI monitors fail: they name
|
|
# a `metrics` port their Service does not have, so they discover zero targets and their
|
|
# `up{job=...}` alerts are inert in the very same way).
|
|
# * no `additionalLabels` needed: platform/monitoring sets serviceMonitorSelectorNilUsesHelmValues
|
|
# false, so Prometheus selects every ServiceMonitor regardless of labels.
|
|
# Cost: one scrape target per manager pod (3), the longhorn_* families — low hundreds of series
|
|
# on this cluster's volume count.
|
|
#
|
|
# PROVEN END TO END ON PROD, 2026-09-13 21:0xZ. The rendered ServiceMonitor was applied to prod
|
|
# by hand (then removed again, so the merge's Helm upgrade creates and OWNS it rather than
|
|
# colliding with an unowned object) with this file's networkpolicy-metrics.yaml already in place:
|
|
# count(up{job="longhorn-backend"}) => 3 (all three managers, up==1)
|
|
# count({__name__=~"longhorn.*"}) => 8717 (was EMPTY)
|
|
# count(longhorn_volume_robustness) => 96 (24 volumes x 4 `state` labels)
|
|
# count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
|
|
# CORRECTION, re-measured 2026-09-13 later the same day: the line originally recorded here read
|
|
# "44 series … 24 healthy(1), 72 unknown(0, detached)". That was a MISREADING of a one-hot
|
|
# metric. `longhorn_volume_robustness` is NOT a 0-3 enum on longhorn 1.12.1; it carries a
|
|
# `state` label (healthy/degraded/faulted/unknown) and its value is only ever 0 or 1. The 72
|
|
# zeros were the other three state labels of the SAME 24 healthy volumes, not 72 detached
|
|
# volumes. The consequence was that enabling this scrape did NOT un-inert
|
|
# LonghornVolumeDegraded/Faulted: their `== 2` / `== 3` expressions still matched no sample.
|
|
# Fixed in platform/monitoring/alerts.yaml and pinned by
|
|
# platform/monitoring/tests/longhorn.test.yaml.
|
|
#
|
|
# Of the rules that WERE evaluating for real, one immediately found a true positive and one
|
|
# permanently-firing defect in the same reading: LonghornVolumeCapacityHigh went pending on the
|
|
# Prometheus TSDB volume at 97.3% ALLOCATED while its filesystem was 20.8% used. Longhorn does
|
|
# not reclaim freed blocks without discard/TRIM, and a 37.7 GiB system snapshot from prod-1's
|
|
# replica rebuild was pinning most of it. That rule has since been DELETED (not re-thresholded):
|
|
# it measured allocation and called it capacity, the filesystem question is already answered by
|
|
# kubelet_volume_stats_*, and the allocation question now lives per disk as
|
|
# LonghornDiskAllocationHigh / LonghornDiskUnschedulable.
|
|
metrics:
|
|
serviceMonitor:
|
|
enabled: true
|
|
|
|
# Longhorn's own manager/UI stay in-cluster; UI exposure (if any) is gated behind
|
|
# oauth2-proxy forward-auth (ADR-0014) by WS-SEC — not exposed here.
|
|
|
|
# TODO(WS-STORAGE, Phase 4): off-cluster backup target (defaultSettings.backupTarget +
|
|
# backupTargetCredentialSecret) is configured alongside Velero (ADR-0003, R-STATE-03).
|
|
# Credentials referenced BY NAME only, SOPS-encrypted (WS-SEC) — never inline here.
|