jakman85andClaude Opus 5
b7341b329b
Three corrections after rebasing onto main. 1. THE GUARD CAUGHT A REAL ONE ON ITS FIRST CONTACT WITH main. main had gained `CNPGPrimariesCoLocated` (platform/cnpg/alerts.yaml) pointing at /runbooks/cnpg-primary-spread, with no runbook. The check failed the rebase exactly as designed — 46 annotations, 35 names, 34 present — naming both the missing file AND the missing configMapGenerator entry. runbooks/cnpg-primary-spread.md now exists: what co-located primaries actually cost here (blast radius, not an outage, because instances: 3 + required anti-affinity), why the drift is uncorrectable-by-declaration (CNPG has no primary-placement field), the readiness gate that keeps the rule quiet during a real node outage, and the ~9s-per-database switchover rebalance, one cluster at a time. The 2026-09-07 outage in which all three primaries sat on one node is the reason the alert exists at all. 2. WHOLE-REPO SCOPE, NOT A HAND-WRITTEN STAGE. main's refactor replaced ci/l0.sh's per-check "UNCONDITIONAL" stages with a declaration: `conformance.home.k3s/scope: "whole-repo"`, unioned into every gate run by Stage 3b's `--include-scope`. My +38-line Stage 3f is removed (ci/l0.sh is now byte-identical to main) and the annotation is declared instead. Assertion (7) of the check was rewritten to match: it no longer asserts its own path appears in ci/l0.sh — it asserts the test DECLARES the scope and that ci/l0.sh still passes --include-scope, since neither half alone makes the check load-bearing. That matters here because ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02", not R-OBS-04, and maps platform/kubevirt/* and runbooks/* to nothing at all. 3. A FACTUAL CORRECTION IN ALL 35 RUNBOOKS. Every page claimed `dev.sh` defaults its current-context to k3s-dev. main's `9b4744d fix(devcontainer): dev.sh defaults to prod` made that false, and a runbook that misstates which cluster a command talks to is worse than one that says nothing. The pages now say what is durably true: the default has already changed once, so naming `--context k3s-prod` is what makes a command mean the same thing next month, and the per-invocation banner is there to be read. The check's own failure message was corrected the same way. Re-verified after all three: check green at 374 assertions (46 annotations, 35/35 runbooks present, 35/35 projected), re-applied to prod, and re-probed in-cluster through the Gateway — node-down, keycloak-down, keycloak-client-secrets, cnpg-archiving, longhorn-faulted, vpn-down, node-io and cnpg-primary-spread all HTTP 200 with text/plain; charset=utf-8 and 8.9-12.1 KB of real body; /runbooks/ index 200; an unknown name 404s with a pointer rather than a blank page; and homepage still answers the apex `/`, confirming Gateway API longest-path precedence. Full merge gate: PASS (spec-immutability, lint, l0, render-golden, l1 all green; prom-rules SKIPPED for want of promtool in the image; L2/L3 SKIPPED, no cluster). Co-Authored-By: Claude Opus 5 <[email protected]>
184 lines
10 KiB
Markdown
184 lines
10 KiB
Markdown
# node-io — Disk I/O pressure / starvation on a node
|
|
|
|
**Alerts:** `NodeIOPressureHigh` (warning, 10m) and `NodeIOPressureCritical` (critical, 5m)
|
|
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes`
|
|
**Expression:** `rate(node_pressure_io_stalled_seconds_total[5m]) > 0.05` (warning) / `> 0.10` (critical)
|
|
|
|
## What this means here
|
|
|
|
PSI **full** for I/O: the fraction of wall-clock time in which *every* runnable task on the node was
|
|
blocked on disk, i.e. the machine did no useful work. It is not "the disk is busy" — it is "the node
|
|
is stalled".
|
|
|
|
This alert exists because of a real outage it would have caught and could not. On **2026-09-07 at
|
|
06:05:21Z, `k3s-prod-1` went NotReady** ("Kubelet stopped posting node status") while the host stayed
|
|
up, pingable and SSH-able. The k3s journal showed embedded etcd starved of disk:
|
|
|
|
```
|
|
slow fdatasync took=3.56s / 6.71s / 4.95s / 3.40s (etcd expects 1s)
|
|
apply request took too long took=8.30s (expects 100ms)
|
|
etcdserver: too many requests
|
|
Failed to check local etcd status for learner management: context deadline exceeded
|
|
```
|
|
|
|
`/proc/pressure/io` **full avg300** at the time: `k3s-prod-1` **11.36**, `k3s-prod-2` (PM9A1 NVMe)
|
|
**0.75**, `k3s-prod-3` (WD Blue) **1.29**, against a ~1% steady state. The stack alerted on CPU-PSI
|
|
and memory and had **no I/O alert at all**, so the first signal the operator got was a node already
|
|
out of the cluster. The thresholds here are those measurements, not guesses: 0.05 is 5x steady
|
|
state, 0.10 is the level node 1 actually held while etcd stalled.
|
|
|
|
**Why this is self-reinforcing, and why it is urgent.** On every node the etcd WAL (`/data/k3s`) and
|
|
all Longhorn replicas (`/data/longhorn`) sit on **one partition** — `node_baseline_etcd_source:
|
|
shared-data-partition`, not a dedicated device, and §2's exit condition in
|
|
`infra/PROD-PROVISIONING.md` is still **open**. So: node stalls → drops out → its volumes go degraded
|
|
→ Longhorn rebuilds them → more writes to the same spindle etcd fsyncs to → worse. Recovery on
|
|
2026-09-07 needed a manual `systemctl restart k3s`, and I/O pressure only fell when the 18 rebuilds
|
|
finished.
|
|
|
|
**There is no etcd fsync metric to check.** `etcd_disk_wal_fsync_duration_seconds` is not collected —
|
|
`kubeEtcd` is deliberately disabled in `platform/monitoring/kube-prometheus-stack.yaml` for k3s. PSI
|
|
is the only signal that exists. Do not go looking for the etcd dashboard; it is not there.
|
|
|
|
## Running the commands in this runbook
|
|
|
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
|
that hang is not the cluster being down. Every command below runs as:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
|
|
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
|
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
|
|
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
|
|
the same thing next month as it does today; one that relies on the default does not, and the failure is
|
|
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
|
|
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
|
|
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
|
|
interfere.
|
|
|
|
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
|
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
|
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
|
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
|
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
|
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
|
|
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
|
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
|
Where this runbook gives a mutating command it is for an operator, and it says so.
|
|
|
|
## First commands
|
|
|
|
Which node, and how bad:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get nodes -o wide'
|
|
```
|
|
|
|
Read PSI and per-device queue depth on the node itself (SSH direct — prod is on
|
|
`192.168.2.0/24` with no NAT hop; `.10` = prod-1, `.11` = prod-2, `.12` = prod-3):
|
|
|
|
```
|
|
ssh k3s-prod-1 'cat /proc/pressure/io; echo; iostat -x 1 3; echo; lsblk -o NAME,MOUNTPOINT,ROTA'
|
|
```
|
|
|
|
Is etcd already stalling? This is the line that decides urgency:
|
|
|
|
```
|
|
ssh k3s-prod-1 'journalctl -u k3s --since "30 min ago" | grep -Ei "fdatasync|apply request took|too many requests|leader"'
|
|
```
|
|
|
|
Is Longhorn rebuilding (the usual writer)?
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
|
|
-o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,NODE:.status.currentNodeID | grep -v healthy'
|
|
```
|
|
|
|
Is the disk itself dying? (This is what it was in September.)
|
|
|
|
```
|
|
ssh k3s-prod-1 'sudo smartctl -a /dev/sda | grep -Ei "Reallocated|Pending|Uncorrectable|Wear|Power_On_Hours|Model"'
|
|
```
|
|
|
|
## Likely causes, most likely first
|
|
|
|
### 1. A Longhorn replica rebuild storm (most common by far)
|
|
Every node reboot degrades all ~18 volumes and triggers a full rebuild. That is the normal cost of
|
|
Longhorn on these disks, and it is why reboots are rolled **one node at a time, waiting for
|
|
`robustness=healthy` on every volume before the next**.
|
|
|
|
Remediation — freeze rebuilds while the node recovers (operator, mutating):
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n longhorn-system patch settings.longhorn.io \
|
|
concurrent-replica-rebuild-per-node-limit --type=merge -p "{\"value\":\"0\"}"'
|
|
```
|
|
|
|
Thaw to `"1"` — never higher — once all volumes read `healthy`. Use `0` when replicas will
|
|
**reattach** (an in-place bounce); use `1` when they are genuinely gone (an OS re-provision).
|
|
`concurrentReplicaRebuildPerNodeLimit: 1` was **already in force during the 2026-09-07 outage and
|
|
held** — do not "fix" this by setting a knob that is already set.
|
|
|
|
### 2. The node's disk is failing
|
|
`k3s-prod-1`'s original 2011 `C400-MTFDDAC512MAM` had **61,440 reallocated sectors**, 31% endurance
|
|
consumed and no power-loss protection. The operator replaced it with a Samsung 850 EVO on 2026-09-07
|
|
and the same 18-replica rebuild measured **3.79** instead of 11.36 — ~3x better, node stayed Ready.
|
|
The earlier dev-era drive (Seagate `ST4000DM000`: 29,408 reallocated, 448 pending, 448 offline
|
|
uncorrectable) drove etcd to 48% iowait. If SMART shows reallocated or pending sectors climbing,
|
|
**the fix is hardware**; nothing in software recovers it.
|
|
|
|
When replacing a disk, two traps recorded from that rebuild:
|
|
- Longhorn refuses the new disk with `DiskFilesystemChanged` ("record diskUUID doesn't match the one
|
|
on the disk"). Remedy: **disable → remove the disk entry → re-add, in that order** (the validating
|
|
webhook enforces it).
|
|
- `inventory/hosts.ini` decides `--cluster-init` via `[k3s_prod_primary]`. Re-provisioning a node
|
|
that is still listed there **founds a second cluster**. The founder now sits on `k3s-prod-2`.
|
|
|
|
### 3. A Velero backup or restore running against the same spindle
|
|
Velero's daily runs at `0 3 * * *` and reads Longhorn PVCs through kopia. Check
|
|
`backups.velero.io` (fully qualified — see above) for one in flight, and the Longhorn limits
|
|
`concurrentVolumeBackupRestorePerNodeLimit` / `snapshotHeavyTaskConcurrentLimit`, both pinned to 1
|
|
for exactly this reason.
|
|
|
|
### 4. A CDI image import or a TCG-emulated VM
|
|
A CDI import writes a whole disk image; heavy sustained node load from TCG-emulated VMs has also been
|
|
observed to degrade the Cilium datapath into intermittent LB-VIP unreachability (cleared by a
|
|
restart). Check `kubectl --context k3s-prod get vmi -A` and importer pods in `cdi`.
|
|
|
|
### 5. Structural: etcd and Longhorn share one partition
|
|
This is the standing cause underneath the other four, and it is open work. `roles/etcd_io_priority`
|
|
+ `playbooks/etcd-io-priority.yml` apply BFQ on the shared disk, `nr_requests` 64→32 and
|
|
`IOWeight=10000` on `system.slice` and `k3s.service`, reading every value back. **It applies nothing
|
|
until an operator runs it** — run `k3s-prod-3` first, with `--limit` and `--list-hosts`.
|
|
|
|
Four mechanisms are **proven inert on these nodes** — do not re-try them:
|
|
`IOSchedulingClass=realtime` on `k3s.service` is a footgun (ioprio is inherited across fork/exec and
|
|
containerd is a child of the k3s server process, so realtime for k3s means realtime for Longhorn
|
|
too); `IOSchedulingPriority` is a no-op under `mq-deadline`; `io.latency` is unavailable
|
|
(`CONFIG_BLK_CGROUP_IOLATENCY` is not built in Debian trixie); a bare `IOWeight=` drop-in is
|
|
decoration (blk-iocost compiled-but-disabled, BFQ not loaded). And no cgroup or scheduler control
|
|
arbitrates the **jbd2 journal** that etcd's `fdatasync` actually waits on.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **Stop and get the operator** if `NodeIOPressureCritical` is firing *and* the journal shows
|
|
`slow fdatasync` or `apply request took too long`. The node is minutes from leaving the cluster.
|
|
With one of three etcd members already gone, a second loss ends quorum.
|
|
- **Do not reboot or drain a second node** while any volume is not `healthy`. That is how a
|
|
single-node problem becomes a data problem on a 3-replica cluster.
|
|
- **Do not `kubectl drain`.** This cluster does not drain: the posture is deliberate (the upgrade
|
|
plan rejects system-upgrade-controller because "its auto-drain fights this cluster's slow-HDD
|
|
Longhorn"), and the node-loss conformance test evicts only its own writer pod.
|
|
- Hand over to the operator for anything physical (disk replacement) or for the Ansible role — both
|
|
are outside what can be done from git.
|
|
|
|
---
|
|
Source of truth: `runbooks/node-io.md` in `home-network-k3s-impl`. Index of all runbooks: `/runbooks/`
|
|
Incident record: `DECISIONS-LOG.md` "2026-09-07 — k3s-prod-1 NotReady: etcd starved of disk I/O by
|
|
Longhorn", and `infra/PROD-PROVISIONING.md` §0c / §2.
|