jakman85andClaude Opus 5
8e017fad91
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
213 lines
12 KiB
Markdown
213 lines
12 KiB
Markdown
# longhorn-capacity — A Longhorn disk is running out of room for replicas
|
||
|
||
**Alerts:** `LonghornDiskAllocationHigh` (warning, 30m) and `LonghornDiskUnschedulable` (warning, 15m)
|
||
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.longhorn`
|
||
**Expressions:**
|
||
`(longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70`
|
||
`longhorn_disk_status{condition="schedulable"} == 0`
|
||
|
||
## What this means here
|
||
|
||
**This is a POOL question, not a PVC question.** One of the three nodes' Longhorn data disks is
|
||
filling with replica files, or Longhorn has already stopped placing replicas on it. Neither alert
|
||
says anything about how full any single workload's filesystem is — for that, read
|
||
`/runbooks/pvc-capacity` (`PersistentVolumeFillingUp`, `PersistentVolumeAlmostFull`) or
|
||
`/runbooks/monitoring-capacity` (`MonitoringVolumeFillingUp`). Those are keyed on
|
||
`kubelet_volume_stats_*`, which is the filesystem's own accounting.
|
||
|
||
`longhorn_disk_usage_bytes` counts bytes ALLOCATED: every replica file plus every snapshot still
|
||
pinned to it. Longhorn does not return freed blocks to a replica file without discard/TRIM, and a
|
||
system snapshot taken for a replica rebuild keeps its historical blocks alive, so this number only
|
||
goes up until something is deleted. That is the correct question *here* — allocated bytes are what
|
||
consumes the node's ~414 GiB data partition — and the wrong question per volume, which is why
|
||
`LonghornVolumeCapacityHigh` was deleted on 2026-09-13 (it read 0.973 on a Prometheus TSDB volume
|
||
whose filesystem was 20.8% used, by design, permanently). See the comment block in
|
||
`platform/monitoring/alerts.yaml` for the measurements.
|
||
|
||
**Why 0.70 and not 0.85.** `storageMinimalAvailablePercentage: 25` means Longhorn itself refuses to
|
||
schedule a new replica on a disk with under 25% available — i.e. at **75% used the disk is already
|
||
out of the scheduling pool**. An 85% threshold could only ever fire after that had happened. 0.70 is
|
||
the last point at which there is still lead time. The companion settings are
|
||
`storageReservedPercentageForDefaultDisk: 20`, `storageOverProvisioningPercentage: 200`,
|
||
`defaultDataLocality: disabled`, hard node anti-affinity.
|
||
|
||
Measured on prod 2026-09-13: usage 0.2254 / 0.2306 / 0.2369 on prod-1/2/3, with 268.9 GiB of nominal
|
||
replica size scheduled against (414.5 GiB − 20% reserved) × 200%. So either alert firing means
|
||
something changed materially, not that the platform slowly grew.
|
||
|
||
The severe consequence is that a degraded volume **cannot finish rebuilding** — hard anti-affinity
|
||
needs three distinct nodes with room, so one full or unschedulable disk pins a volume at 2 replicas
|
||
indefinitely, and R-STATE-02's survive-a-node-loss promise is suspended for as long as that lasts.
|
||
Cross-read `/runbooks/longhorn-degraded`.
|
||
|
||
### Verify the alert can fire (measured 2026-09-13)
|
||
|
||
These alerts were INERT until 2026-09-13, when `metrics.serviceMonitor.enabled` was set in
|
||
`platform/longhorn/helmrelease.yaml`; the longhorn_* family was simply absent from the TSDB before
|
||
that. Both are now live — verified on prod the same day:
|
||
|
||
```
|
||
infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \
|
||
prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
|
||
/bin/promtool query instant http://localhost:9090 \
|
||
'count(longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) + count(longhorn_disk_status{condition="schedulable"})'
|
||
```
|
||
|
||
returned **6** (3 disks, both selectors resolving), while both alert expressions returned empty.
|
||
If that count is ever empty again, the scrape has broken and this pair is silently back to inert —
|
||
check `up{job="longhorn-backend"}` and `platform/longhorn/networkpolicy-metrics.yaml`.
|
||
|
||
## Running the commands in this runbook
|
||
|
||
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
||
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
||
that hang is not the cluster being down. Every command below runs as:
|
||
|
||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
||
|
||
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
||
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
|
||
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
|
||
the same thing next month as it does today; one that relies on the default does not, and the failure is
|
||
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
|
||
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
|
||
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
|
||
interfere.
|
||
|
||
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
||
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
||
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
||
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
||
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
||
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||
|
||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||
and it says so.
|
||
|
||
## First commands
|
||
|
||
Per-disk allocation and Longhorn's own scheduling verdict — the two things the alerts read:
|
||
|
||
```
|
||
infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \
|
||
prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
|
||
/bin/promtool query instant http://localhost:9090 \
|
||
'longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes or longhorn_disk_status'
|
||
```
|
||
|
||
Node-level allocation, from the CRs:
|
||
|
||
```
|
||
infra/devcontainer/dev.sh run bash -c \
|
||
'kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o yaml | grep -E "name:|storageAvailable|storageMaximum|storageScheduled|allowScheduling"'
|
||
```
|
||
|
||
Volume nominal size versus ALLOCATED size. `actualSize` is head + pinned snapshots, NOT filesystem
|
||
usage — a volume can read near 100% here with a nearly empty filesystem (see the deleted-alert note
|
||
above), so do not read this column as "full":
|
||
|
||
```
|
||
infra/devcontainer/dev.sh run bash -c \
|
||
'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
|
||
-o custom-columns=NAME:.metadata.name,SPEC:.spec.size,ACTUAL:.status.actualSize,ROBUST:.status.robustness'
|
||
```
|
||
|
||
Which PVC each one is, so you know who is filling it:
|
||
|
||
```
|
||
infra/devcontainer/dev.sh run bash -c \
|
||
'kubectl --context k3s-prod get pv -o custom-columns=PV:.metadata.name,CLAIM:.spec.claimRef.namespace/.spec.claimRef.name,SIZE:.spec.capacity.storage,SC:.spec.storageClassName'
|
||
```
|
||
|
||
Which snapshots are holding the allocation (this is the measured top cause on prod — one
|
||
`user_created=false` system snapshot on the Prometheus TSDB volume held 37.7 GiB on 2026-09-13):
|
||
|
||
```
|
||
infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \
|
||
prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
|
||
/bin/promtool query instant http://localhost:9090 \
|
||
'topk(10, longhorn_snapshot_actual_size_bytes)'
|
||
```
|
||
|
||
The filesystem view — the separate question, answered by `/runbooks/pvc-capacity`:
|
||
|
||
```
|
||
infra/devcontainer/dev.sh run bash -c \
|
||
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
|
||
curl -s -G --data-urlencode "query=topk(10, 1 - kubelet_volume_stats_available_bytes{job=\"kubelet\"} / kubelet_volume_stats_capacity_bytes{job=\"kubelet\"})" localhost:19090/api/v1/query'
|
||
```
|
||
|
||
On the node itself:
|
||
|
||
```
|
||
ssh k3s-prod-1 'df -h /data; du -xh --max-depth=1 /data/longhorn/replicas 2>/dev/null | sort -h | tail'
|
||
```
|
||
|
||
## Likely causes, most likely first
|
||
|
||
### 0. Pinned snapshots, not live data (check this FIRST for an allocation alert)
|
||
An allocation alert can fire with every filesystem in the cluster nearly empty. Run the
|
||
`longhorn_snapshot_actual_size_bytes` query above before anything else. A replica rebuild — which is
|
||
what a node reboot or a disk replacement forces, and prod-1's drive was replaced on 2026-09-12 —
|
||
leaves a `user_created=false` system snapshot behind, and it keeps its blocks until it is purged.
|
||
Prune from the Longhorn UI (Volume → Snapshots), then re-read the query; nothing else is needed if
|
||
this was the cause.
|
||
|
||
### 1. A CNPG database is not recycling WAL (check this FIRST for a PVC-fill alert)
|
||
The only cause here that grows without bound. `archive_mode=on` plus a failing `archive_command` means
|
||
Postgres will not recycle unarchived WAL; measured on prod `keycloak-db` at ~6.8 MiB/h with `pg_wal`
|
||
sharing the data PVC (no `walStorage:` is configured on any Cluster) — about 29 days from 93% free to
|
||
disk-full, ending in a postmaster PANIC. Go to `/runbooks/cnpg-archiving`. **Never delete WAL by
|
||
hand.**
|
||
|
||
### 2. Snapshots and orphaned replicas accumulating
|
||
Longhorn snapshots are not free, and cleanup here is **manual**. `orphanAutoDeletion: true` used to
|
||
sit in `platform/longhorn/helmrelease.yaml` and had been **silently inert since Longhorn 1.12** — it
|
||
was renamed `orphanResourceAutoDeletion` (now a list) and the chart drops unknown `defaultSettings`
|
||
keys with no error. It was **removed on 2026-09-13** and deliberately not replaced, so orphan cleanup
|
||
is manual by decision, not by accident: prune orphans and stale snapshots from the Longhorn UI.
|
||
|
||
Note also that **snapshot purge and clone now run one at a time per node**
|
||
(`snapshotHeavyTaskConcurrentLimit: 1`, declared 2026-09-13 to keep Longhorn from starving etcd's WAL
|
||
fsync). Reclaim after a rebuild storm is therefore *serial and slower by design*. If allocation is
|
||
climbing and purge is the bottleneck, the operator may raise the Setting for the duration and watch
|
||
`NodeIOPressureHigh` — see `runbooks/longhorn-degraded.md` for the tradeoff as recorded.
|
||
|
||
### 3. A workload genuinely grew
|
||
Plex/Sonarr/Radarr **media** does not live on Longhorn — it is static NFS from
|
||
`192.168.2.102:/mnt/Storage/plex`, via the `bulk` tier. If a media volume is on `longhorn`, that is a
|
||
misconfiguration worth fixing, not a capacity problem: the `bulk` StorageClass is a stub
|
||
(`provisioner: nfs.csi.k8s.io`, `server:/share: REPLACE-ME`) and provisions nothing, so real bulk
|
||
storage is static NFS PVs. Otherwise expand the PVC — all Longhorn classes have
|
||
`allowVolumeExpansion: true`; change the size in git and push.
|
||
|
||
### 4. Replicas piled onto one node
|
||
`replicaAutoBalance: best-effort` should prevent this, and hard anti-affinity bounds it to one replica
|
||
per volume per node. If one node is much fuller than the others, check whether a rebuild placed
|
||
replicas there while another node was down.
|
||
|
||
### 5. The cluster has outgrown its disks
|
||
Prod's ceiling is ~829 GiB across three nodes. `docs/plans/longhorn-ssd-tiering.md` is the planned
|
||
answer (a ~51 GB hot tier: Prometheus, CNPG, Loki, Gitea). Operator planning work.
|
||
|
||
## When to escalate or stop
|
||
|
||
- **A disk over 70% while any volume is degraded is urgent**, because the rebuild cannot complete —
|
||
and at 75% Longhorn takes the disk out of the scheduling pool entirely
|
||
(`LonghornDiskUnschedulable`). Free space or restore the other node before anything else.
|
||
- **Never delete files under `/data/longhorn`** or inside a Postgres `pg_wal`. Both are irreversible.
|
||
- **Do not expand a PVC as a reflex** if the cause is unrecycled WAL — you are buying days and hiding
|
||
the real fault.
|
||
- Escalate for adding disks, the SSD tiering plan, or anything that needs a node reboot (each reboot
|
||
degrades all ~18 volumes and triggers a rebuild storm).
|
||
|
||
---
|
||
Source of truth: `runbooks/longhorn-capacity.md`. Index: `/runbooks/`
|
||
Records: `platform/longhorn/helmrelease.yaml` (the settings above, the I/O throttle block, and the
|
||
removal of the inert `orphanAutoDeletion`);
|
||
`platform/cnpg/alerts.yaml` header (the WAL disk-full clock); `docs/BULK-NFS-STORAGE.md`.
|