jakman85andClaude Opus 5
8e017fad91
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
135 lines
7.8 KiB
Markdown
135 lines
7.8 KiB
Markdown
# node-disk — A node filesystem is filling up
|
|
|
|
**Alert:** `NodeDiskFillingUp` (warning, 15m)
|
|
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes`
|
|
**Expression:** `node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"} / node_filesystem_size_bytes{...} < 0.10`
|
|
|
|
## What this means here
|
|
|
|
Less than 10% free on a real filesystem on a node, for 15 minutes. Read the `mountpoint` label
|
|
first, because on these nodes it tells you which of two very different problems you have.
|
|
|
|
**`/data` is the one that matters.** Both the embedded **etcd WAL** (`/data/k3s`) and **every
|
|
Longhorn replica** (`/data/longhorn`) live on that single partition —
|
|
`node_baseline_etcd_source: shared-data-partition`, not a dedicated device, and the exit condition
|
|
for splitting them is still open (`infra/PROD-PROVISIONING.md` §2). If `/data` fills:
|
|
|
|
- etcd cannot write its WAL. A control-plane member with a full WAL partition does not degrade
|
|
gracefully; it stops, and this is a 3-member cluster.
|
|
- Longhorn stops being able to hold replicas and volumes go degraded, which triggers rebuilds, which
|
|
need *more* space.
|
|
|
|
So `/data` at 10% free is not a warning, it is the last comfortable moment to act.
|
|
|
|
**`/` filling up** is usually logs, container images or a stuck journal. Lower stakes, but there is
|
|
history: in the dev era the host root filled to ~99% because VM disks sat under a snapper-snapshotted
|
|
subvolume, and the hypervisor **paused the k3s VMs on I/O error**. Recovery was deleting 11 snapper
|
|
timeline snapshots plus `virsh destroy` on the paused VMs (which were holding rsync-unlinked inodes):
|
|
99% → 30%.
|
|
|
|
Longhorn's own guard rails are already tight and are why this alert usually fires before anything
|
|
breaks: `storageReservedPercentageForDefaultDisk: 20`,
|
|
`storageMinimalAvailablePercentage: 25`, `storageOverProvisioningPercentage: 200`.
|
|
## Running the commands in this runbook
|
|
|
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
|
that hang is not the cluster being down. Every command below runs as:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
|
|
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
|
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
|
|
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
|
|
the same thing next month as it does today; one that relies on the default does not, and the failure is
|
|
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
|
|
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
|
|
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
|
|
interfere.
|
|
|
|
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
|
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
|
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
|
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
|
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
|
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
|
|
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
|
and it says so.
|
|
|
|
## First commands
|
|
|
|
Which filesystem, on which node:
|
|
|
|
```
|
|
ssh k3s-prod-1 'df -h -x tmpfs -x overlay; echo; du -xh --max-depth=1 /data 2>/dev/null | sort -h | tail -10; echo; du -xh --max-depth=1 / 2>/dev/null | sort -h | tail -10'
|
|
```
|
|
|
|
What Longhorn thinks it has:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o custom-columns=NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,SCHED:.spec.allowScheduling; echo; \
|
|
kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,SIZE:.spec.size,ACTUAL:.status.actualSize,ROBUST:.status.robustness'
|
|
```
|
|
|
|
Journal and image bloat on `/`:
|
|
|
|
```
|
|
ssh k3s-prod-1 'sudo journalctl --disk-usage; sudo crictl images | wc -l; sudo du -sh /var/lib/rancher/k3s/agent/containerd 2>/dev/null'
|
|
```
|
|
|
|
## Likely causes, most likely first
|
|
|
|
### 1. Longhorn replicas grew, or replicas landed here after a rebuild
|
|
Hard node anti-affinity (`replicaSoftAntiAffinity: false`) means one replica per node per volume, but
|
|
`replicaAutoBalance: best-effort` moves them. Prod sits around 269 GiB scheduled against an 829 GiB
|
|
ceiling, so this is rarely tight — but a rebuild storm plus snapshots can spike it.
|
|
Remediation: check for orphaned replicas and stale snapshots in the Longhorn UI. Orphan cleanup is
|
|
**manual**: `orphanAutoDeletion: true` sat in `platform/longhorn/helmrelease.yaml` and had been
|
|
**silently inert since Longhorn 1.12** — renamed `orphanResourceAutoDeletion` (a list), and the chart
|
|
drops unknown `defaultSettings` keys with no error. Removed 2026-09-13 and deliberately not replaced.
|
|
Snapshot purge also runs one at a time per node now (`snapshotHeavyTaskConcurrentLimit: 1`, the etcd
|
|
WAL protection), so reclaim after a rebuild is serial by design.
|
|
|
|
### 2. A CNPG database is not recycling WAL
|
|
The single most dangerous cause, because it is unbounded. With `archive_mode=on` and a failing
|
|
`archive_command`, Postgres **refuses to recycle** unarchived WAL segments and `pg_wal` grows forever
|
|
— on prod's `keycloak-db` that was measured at ~6.8 MiB/h, 17 segments / ~268 MiB, about 29 days to
|
|
fill a 5 GiB PVC. `pg_wal` shares the data PVC (no `walStorage:` is configured on any Cluster).
|
|
Check `/runbooks/cnpg-archiving` **before** you free any space, and **never delete WAL by hand** —
|
|
that destroys the database.
|
|
|
|
### 3. Journal / container images on `/`
|
|
`journalctl --vacuum-size=200M` and pruning unused images are safe. Do not prune images for
|
|
workloads that may need to restart while the registry is unreachable.
|
|
|
|
### 4. A Velero or CDI staging write
|
|
Velero's kopia uploads and CDI's scratch space both stage locally. `longhorn-scratch`
|
|
(single-replica) exists precisely because the 3-replica default failed the UniFi 60Gi import on
|
|
insufficient storage.
|
|
|
|
### 5. The disk is smaller than the platform now needs
|
|
`docs/plans/longhorn-ssd-tiering.md` holds the planned hot tier (~51 GB: Prometheus, CNPG, Loki,
|
|
Gitea). Working set measured ~79 GB against ~323 GB free on SSD, so the "won't fit" blocker was a
|
|
measurement artifact. This is operator planning work, not an incident fix.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **If the mountpoint is `/data` and free space is under 5%, treat it as an imminent control-plane
|
|
outage** and get the operator. etcd and Longhorn are both on it.
|
|
- **Never delete anything under `/data/longhorn` or a Postgres `pg_wal` directory by hand.** Both are
|
|
irreversible. Free space elsewhere, or fix the cause.
|
|
- **Do not `kubectl drain`** the node to move load off it — this cluster does not drain.
|
|
- If the cause is a CNPG archive failure, this runbook is the wrong one: go to
|
|
`/runbooks/cnpg-archiving`, which is the actual fix.
|
|
|
|
---
|
|
Source of truth: `runbooks/node-disk.md`. Index: `/runbooks/`
|
|
Records: `DECISIONS-LOG.md` "Disk/drive incident + k3s relocated to NVMe (2026-08-22)";
|
|
`platform/cnpg/alerts.yaml` header (the WAL disk-full clock).
|