0 Fork 0

Files

T

jakman85andClaude Opus 5 b7341b329b fix(runbooks): declare whole-repo scope, drop the bespoke l0 stage, +cnpg-primary-spread

Three corrections after rebasing onto main.

1. THE GUARD CAUGHT A REAL ONE ON ITS FIRST CONTACT WITH main. main had gained
   `CNPGPrimariesCoLocated` (platform/cnpg/alerts.yaml) pointing at
   /runbooks/cnpg-primary-spread, with no runbook. The check failed the rebase exactly as
   designed — 46 annotations, 35 names, 34 present — naming both the missing file AND the
   missing configMapGenerator entry. runbooks/cnpg-primary-spread.md now exists: what
   co-located primaries actually cost here (blast radius, not an outage, because
   instances: 3 + required anti-affinity), why the drift is uncorrectable-by-declaration
   (CNPG has no primary-placement field), the readiness gate that keeps the rule quiet
   during a real node outage, and the ~9s-per-database switchover rebalance, one cluster
   at a time. The 2026-09-07 outage in which all three primaries sat on one node is the
   reason the alert exists at all.

2. WHOLE-REPO SCOPE, NOT A HAND-WRITTEN STAGE. main's refactor replaced ci/l0.sh's
   per-check "UNCONDITIONAL" stages with a declaration:
   `conformance.home.k3s/scope: "whole-repo"`, unioned into every gate run by Stage 3b's
   `--include-scope`. My +38-line Stage 3f is removed (ci/l0.sh is now byte-identical to
   main) and the annotation is declared instead. Assertion (7) of the check was rewritten
   to match: it no longer asserts its own path appears in ci/l0.sh — it asserts the test
   DECLARES the scope and that ci/l0.sh still passes --include-scope, since neither half
   alone makes the check load-bearing. That matters here because
   ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02", not
   R-OBS-04, and maps platform/kubevirt/* and runbooks/* to nothing at all.

3. A FACTUAL CORRECTION IN ALL 35 RUNBOOKS. Every page claimed `dev.sh` defaults its
   current-context to k3s-dev. main's `9b4744d fix(devcontainer): dev.sh defaults to
   prod` made that false, and a runbook that misstates which cluster a command talks to is
   worse than one that says nothing. The pages now say what is durably true: the default
   has already changed once, so naming `--context k3s-prod` is what makes a command mean
   the same thing next month, and the per-invocation banner is there to be read. The
   check's own failure message was corrected the same way.

Re-verified after all three: check green at 374 assertions (46 annotations, 35/35 runbooks
present, 35/35 projected), re-applied to prod, and re-probed in-cluster through the
Gateway — node-down, keycloak-down, keycloak-client-secrets, cnpg-archiving,
longhorn-faulted, vpn-down, node-io and cnpg-primary-spread all HTTP 200 with
text/plain; charset=utf-8 and 8.9-12.1 KB of real body; /runbooks/ index 200; an unknown
name 404s with a pointer rather than a blank page; and homepage still answers the apex `/`,
confirming Gateway API longest-path precedence.

Full merge gate: PASS (spec-immutability, lint, l0, render-golden, l1 all green;
prom-rules SKIPPED for want of promtool in the image; L2/L3 SKIPPED, no cluster).

Co-Authored-By: Claude Opus 5 <[email protected]>

2026-09-13 00:00:24 -05:00

9.0 KiB

Raw Blame History

longhorn-capacity — A Longhorn volume or node is near its storage capacity

Alerts: LonghornVolumeCapacityHigh (warning, 30m) and LonghornNodeStorageHigh (warning, 30m) File: platform/monitoring/alerts.yaml, group platform-health.longhorn Expressions: (longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85 (longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes) > 0.85

What this means here

Either one volume is 85% full, or one node's Longhorn disk is 85% allocated. They are different problems with different fixes, so read the labels: volume means a workload is filling its own PVC; node means the cluster is running out of places to put replicas.

The node case is bounded by settings that are already conservative: storageReservedPercentageForDefaultDisk: 20, storageMinimalAvailablePercentage: 25, storageOverProvisioningPercentage: 200, defaultDataLocality: disabled, hard node anti-affinity. Measured headroom on prod is roughly 269 GiB scheduled against an 829 GiB ceiling, so the node alert firing means something has changed materially, not that the platform has slowly grown.

The severe consequence of running out of node storage is that a degraded volume cannot finish rebuilding — hard anti-affinity needs three distinct nodes with room, so one full node pins a volume at 2 replicas indefinitely. Cross-read /runbooks/longhorn-degraded.

Verify the alert can even fire (measured 2026-09-13)

Prometheus on prod scrapes 19 jobs: apiserver, cdi-prometheus-metrics, coredns, keycloak-discovery, keycloak-service, keycloak/keycloak-db, kube-prometheus-stack-alertmanager, kube-prometheus-stack-grafana, kube-prometheus-stack-operator, kube-prometheus-stack-prometheus, kube-state-metrics, kubelet, kubevirt-prometheus-metrics, nextcloud-db/nextcloud-db, node-exporter, oauth2-proxy, semaphore-db/semaphore-db, velero, velero/node-agent. There is no Longhorn, cert-manager, Flux-controller or Hubble/Cilium job. Check the series this rule depends on before you trust its silence:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & \
   sleep 5; curl -s -G --data-urlencode "query=count by (job) (up)" localhost:19090/api/v1/query'

These alerts are currently INERT. count(longhorn_volume_capacity_bytes) and count(longhorn_node_storage_capacity_bytes) both returned 0 series on prod on 2026-09-13 — Longhorn ships no ServiceMonitor here. Volume-fill coverage actually comes from kubelet_volume_stats_* instead, i.e. /runbooks/pvc-capacity, which is where the working alerts are. Use this runbook for the Longhorn-side (node allocation) half, and expect to reach it by hand.

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two dev.sh invocations at once; they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git (git push origin main is the deploy). Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

Node-level allocation (the half that has no substitute alert):

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o yaml | grep -E "name:|storageAvailable|storageMaximum|storageScheduled|allowScheduling"'

Volume sizes versus actual usage:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
     -o custom-columns=NAME:.metadata.name,SPEC:.spec.size,ACTUAL:.status.actualSize,ROBUST:.status.robustness'

Which PVC each one is, so you know who is filling it:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get pv -o custom-columns=PV:.metadata.name,CLAIM:.spec.claimRef.namespace/.spec.claimRef.name,SIZE:.spec.capacity.storage,SC:.spec.storageClassName'

The filesystem view, which is the alert that actually works:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   curl -s -G --data-urlencode "query=topk(10, 1 - kubelet_volume_stats_available_bytes{job=\"kubelet\"} / kubelet_volume_stats_capacity_bytes{job=\"kubelet\"})" localhost:19090/api/v1/query'

On the node itself:

ssh k3s-prod-1 'df -h /data; du -xh --max-depth=1 /data/longhorn/replicas 2>/dev/null | sort -h | tail'

Likely causes, most likely first

1. A CNPG database is not recycling WAL (check this FIRST)

The only cause here that grows without bound. archive_mode=on plus a failing archive_command means Postgres will not recycle unarchived WAL; measured on prod keycloak-db at ~6.8 MiB/h with pg_wal sharing the data PVC (no walStorage: is configured on any Cluster) — about 29 days from 93% free to disk-full, ending in a postmaster PANIC. Go to /runbooks/cnpg-archiving. Never delete WAL by hand.

2. Snapshots and orphaned replicas accumulating

Longhorn snapshots are not free, and cleanup here is manual: orphanAutoDeletion: true in platform/longhorn/helmrelease.yaml has been silently inert since Longhorn 1.12 — it was renamed orphanResourceAutoDeletion (now a list) and the chart drops unknown defaultSettings keys with no error. Verify the key's current state in that file before citing it; then prune orphans and stale snapshots from the Longhorn UI.

3. A workload genuinely grew

Plex/Sonarr/Radarr media does not live on Longhorn — it is static NFS from 192.168.2.102:/mnt/Storage/plex, via the bulk tier. If a media volume is on longhorn, that is a misconfiguration worth fixing, not a capacity problem: the bulk StorageClass is a stub (provisioner: nfs.csi.k8s.io, server:/share: REPLACE-ME) and provisions nothing, so real bulk storage is static NFS PVs. Otherwise expand the PVC — all Longhorn classes have allowVolumeExpansion: true; change the size in git and push.

4. Replicas piled onto one node

replicaAutoBalance: best-effort should prevent this, and hard anti-affinity bounds it to one replica per volume per node. If one node is much fuller than the others, check whether a rebuild placed replicas there while another node was down.

5. The cluster has outgrown its disks

Prod's ceiling is ~829 GiB across three nodes. docs/plans/longhorn-ssd-tiering.md is the planned answer (a ~51 GB hot tier: Prometheus, CNPG, Loki, Gitea). Operator planning work.

When to escalate or stop

A node over 85% while any volume is degraded is urgent, because the rebuild cannot complete. Free space or restore the other node before anything else.
Never delete files under /data/longhorn or inside a Postgres pg_wal. Both are irreversible.
Do not expand a PVC as a reflex if the cause is unrecycled WAL — you are buying days and hiding the real fault.
Escalate for adding disks, the SSD tiering plan, or anything that needs a node reboot (each reboot degrades all ~18 volumes and triggers a rebuild storm).

Source of truth: runbooks/longhorn-capacity.md. Index: /runbooks/ Records: platform/longhorn/helmrelease.yaml (the settings above and the inert orphanAutoDeletion); platform/cnpg/alerts.yaml header (the WAL disk-full clock); docs/BULK-NFS-STORAGE.md.

9.0 KiB Raw Blame History