jakman85andClaude Opus 5
b7341b329b
Three corrections after rebasing onto main. 1. THE GUARD CAUGHT A REAL ONE ON ITS FIRST CONTACT WITH main. main had gained `CNPGPrimariesCoLocated` (platform/cnpg/alerts.yaml) pointing at /runbooks/cnpg-primary-spread, with no runbook. The check failed the rebase exactly as designed — 46 annotations, 35 names, 34 present — naming both the missing file AND the missing configMapGenerator entry. runbooks/cnpg-primary-spread.md now exists: what co-located primaries actually cost here (blast radius, not an outage, because instances: 3 + required anti-affinity), why the drift is uncorrectable-by-declaration (CNPG has no primary-placement field), the readiness gate that keeps the rule quiet during a real node outage, and the ~9s-per-database switchover rebalance, one cluster at a time. The 2026-09-07 outage in which all three primaries sat on one node is the reason the alert exists at all. 2. WHOLE-REPO SCOPE, NOT A HAND-WRITTEN STAGE. main's refactor replaced ci/l0.sh's per-check "UNCONDITIONAL" stages with a declaration: `conformance.home.k3s/scope: "whole-repo"`, unioned into every gate run by Stage 3b's `--include-scope`. My +38-line Stage 3f is removed (ci/l0.sh is now byte-identical to main) and the annotation is declared instead. Assertion (7) of the check was rewritten to match: it no longer asserts its own path appears in ci/l0.sh — it asserts the test DECLARES the scope and that ci/l0.sh still passes --include-scope, since neither half alone makes the check load-bearing. That matters here because ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02", not R-OBS-04, and maps platform/kubevirt/* and runbooks/* to nothing at all. 3. A FACTUAL CORRECTION IN ALL 35 RUNBOOKS. Every page claimed `dev.sh` defaults its current-context to k3s-dev. main's `9b4744d fix(devcontainer): dev.sh defaults to prod` made that false, and a runbook that misstates which cluster a command talks to is worse than one that says nothing. The pages now say what is durably true: the default has already changed once, so naming `--context k3s-prod` is what makes a command mean the same thing next month, and the per-invocation banner is there to be read. The check's own failure message was corrected the same way. Re-verified after all three: check green at 374 assertions (46 annotations, 35/35 runbooks present, 35/35 projected), re-applied to prod, and re-probed in-cluster through the Gateway — node-down, keycloak-down, keycloak-client-secrets, cnpg-archiving, longhorn-faulted, vpn-down, node-io and cnpg-primary-spread all HTTP 200 with text/plain; charset=utf-8 and 8.9-12.1 KB of real body; /runbooks/ index 200; an unknown name 404s with a pointer rather than a blank page; and homepage still answers the apex `/`, confirming Gateway API longest-path precedence. Full merge gate: PASS (spec-immutability, lint, l0, render-golden, l1 all green; prom-rules SKIPPED for want of promtool in the image; L2/L3 SKIPPED, no cluster). Co-Authored-By: Claude Opus 5 <[email protected]>
159 lines
9.0 KiB
Markdown
159 lines
9.0 KiB
Markdown
# longhorn-capacity — A Longhorn volume or node is near its storage capacity
|
|
|
|
**Alerts:** `LonghornVolumeCapacityHigh` (warning, 30m) and `LonghornNodeStorageHigh` (warning, 30m)
|
|
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.longhorn`
|
|
**Expressions:**
|
|
`(longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85`
|
|
`(longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes) > 0.85`
|
|
|
|
## What this means here
|
|
|
|
Either one volume is 85% full, or one node's Longhorn disk is 85% allocated. They are different
|
|
problems with different fixes, so read the labels: `volume` means a workload is filling its own PVC;
|
|
`node` means the cluster is running out of places to put replicas.
|
|
|
|
The node case is bounded by settings that are already conservative:
|
|
`storageReservedPercentageForDefaultDisk: 20`, `storageMinimalAvailablePercentage: 25`,
|
|
`storageOverProvisioningPercentage: 200`, `defaultDataLocality: disabled`, hard node anti-affinity.
|
|
Measured headroom on prod is roughly **269 GiB scheduled against an 829 GiB ceiling**, so the node
|
|
alert firing means something has changed materially, not that the platform has slowly grown.
|
|
|
|
The severe consequence of running out of node storage is that a degraded volume **cannot finish
|
|
rebuilding** — hard anti-affinity needs three distinct nodes with room, so one full node pins a
|
|
volume at 2 replicas indefinitely. Cross-read `/runbooks/longhorn-degraded`.
|
|
### Verify the alert can even fire (measured 2026-09-13)
|
|
|
|
Prometheus on prod scrapes 19 jobs: `apiserver`, `cdi-prometheus-metrics`, `coredns`,
|
|
`keycloak-discovery`, `keycloak-service`, `keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`,
|
|
`kube-prometheus-stack-grafana`, `kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`,
|
|
`kube-state-metrics`, `kubelet`, `kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`,
|
|
`node-exporter`, `oauth2-proxy`, `semaphore-db/semaphore-db`, `velero`, `velero/node-agent`.
|
|
There is **no** Longhorn, cert-manager, Flux-controller or Hubble/Cilium job. Check the series this
|
|
rule depends on before you trust its silence:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & \
|
|
sleep 5; curl -s -G --data-urlencode "query=count by (job) (up)" localhost:19090/api/v1/query'
|
|
```
|
|
|
|
**These alerts are currently INERT.** `count(longhorn_volume_capacity_bytes)` and
|
|
`count(longhorn_node_storage_capacity_bytes)` both returned **0 series** on prod on 2026-09-13 —
|
|
Longhorn ships no ServiceMonitor here. Volume-fill coverage actually comes from
|
|
`kubelet_volume_stats_*` instead, i.e. `/runbooks/pvc-capacity`, which is where the working alerts
|
|
are. Use this runbook for the Longhorn-side (node allocation) half, and expect to reach it by hand.
|
|
## Running the commands in this runbook
|
|
|
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
|
that hang is not the cluster being down. Every command below runs as:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
|
|
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
|
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
|
|
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
|
|
the same thing next month as it does today; one that relies on the default does not, and the failure is
|
|
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
|
|
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
|
|
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
|
|
interfere.
|
|
|
|
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
|
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
|
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
|
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
|
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
|
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
|
|
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
|
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
|
Where this runbook gives a mutating command it is for an operator, and it says so.
|
|
|
|
## First commands
|
|
|
|
Node-level allocation (the half that has no substitute alert):
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o yaml | grep -E "name:|storageAvailable|storageMaximum|storageScheduled|allowScheduling"'
|
|
```
|
|
|
|
Volume sizes versus actual usage:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
|
|
-o custom-columns=NAME:.metadata.name,SPEC:.spec.size,ACTUAL:.status.actualSize,ROBUST:.status.robustness'
|
|
```
|
|
|
|
Which PVC each one is, so you know who is filling it:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get pv -o custom-columns=PV:.metadata.name,CLAIM:.spec.claimRef.namespace/.spec.claimRef.name,SIZE:.spec.capacity.storage,SC:.spec.storageClassName'
|
|
```
|
|
|
|
The filesystem view, which is the alert that actually works:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
|
|
curl -s -G --data-urlencode "query=topk(10, 1 - kubelet_volume_stats_available_bytes{job=\"kubelet\"} / kubelet_volume_stats_capacity_bytes{job=\"kubelet\"})" localhost:19090/api/v1/query'
|
|
```
|
|
|
|
On the node itself:
|
|
|
|
```
|
|
ssh k3s-prod-1 'df -h /data; du -xh --max-depth=1 /data/longhorn/replicas 2>/dev/null | sort -h | tail'
|
|
```
|
|
|
|
## Likely causes, most likely first
|
|
|
|
### 1. A CNPG database is not recycling WAL (check this FIRST)
|
|
The only cause here that grows without bound. `archive_mode=on` plus a failing `archive_command` means
|
|
Postgres will not recycle unarchived WAL; measured on prod `keycloak-db` at ~6.8 MiB/h with `pg_wal`
|
|
sharing the data PVC (no `walStorage:` is configured on any Cluster) — about 29 days from 93% free to
|
|
disk-full, ending in a postmaster PANIC. Go to `/runbooks/cnpg-archiving`. **Never delete WAL by
|
|
hand.**
|
|
|
|
### 2. Snapshots and orphaned replicas accumulating
|
|
Longhorn snapshots are not free, and cleanup here is **manual**: `orphanAutoDeletion: true` in
|
|
`platform/longhorn/helmrelease.yaml` has been **silently inert since Longhorn 1.12** — it was renamed
|
|
`orphanResourceAutoDeletion` (now a list) and the chart drops unknown `defaultSettings` keys with no
|
|
error. Verify the key's current state in that file before citing it; then prune orphans and stale
|
|
snapshots from the Longhorn UI.
|
|
|
|
### 3. A workload genuinely grew
|
|
Plex/Sonarr/Radarr **media** does not live on Longhorn — it is static NFS from
|
|
`192.168.2.102:/mnt/Storage/plex`, via the `bulk` tier. If a media volume is on `longhorn`, that is a
|
|
misconfiguration worth fixing, not a capacity problem: the `bulk` StorageClass is a stub
|
|
(`provisioner: nfs.csi.k8s.io`, `server:/share: REPLACE-ME`) and provisions nothing, so real bulk
|
|
storage is static NFS PVs. Otherwise expand the PVC — all Longhorn classes have
|
|
`allowVolumeExpansion: true`; change the size in git and push.
|
|
|
|
### 4. Replicas piled onto one node
|
|
`replicaAutoBalance: best-effort` should prevent this, and hard anti-affinity bounds it to one replica
|
|
per volume per node. If one node is much fuller than the others, check whether a rebuild placed
|
|
replicas there while another node was down.
|
|
|
|
### 5. The cluster has outgrown its disks
|
|
Prod's ceiling is ~829 GiB across three nodes. `docs/plans/longhorn-ssd-tiering.md` is the planned
|
|
answer (a ~51 GB hot tier: Prometheus, CNPG, Loki, Gitea). Operator planning work.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **A node over 85% while any volume is degraded is urgent**, because the rebuild cannot complete.
|
|
Free space or restore the other node before anything else.
|
|
- **Never delete files under `/data/longhorn`** or inside a Postgres `pg_wal`. Both are irreversible.
|
|
- **Do not expand a PVC as a reflex** if the cause is unrecycled WAL — you are buying days and hiding
|
|
the real fault.
|
|
- Escalate for adding disks, the SSD tiering plan, or anything that needs a node reboot (each reboot
|
|
degrades all ~18 volumes and triggers a rebuild storm).
|
|
|
|
---
|
|
Source of truth: `runbooks/longhorn-capacity.md`. Index: `/runbooks/`
|
|
Records: `platform/longhorn/helmrelease.yaml` (the settings above and the inert `orphanAutoDeletion`);
|
|
`platform/cnpg/alerts.yaml` header (the WAL disk-full clock); `docs/BULK-NFS-STORAGE.md`.
|