# node-cpu — CPU pressure or run-queue depth high on a node **Alerts:** `NodeCPUPressureHigh` (warning, 15m) and `NodeCPULoadHigh` (warning, 15m) **File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes` **Expressions:** `rate(node_pressure_cpu_waiting_seconds_total[5m]) > 0.15` `node_load5 / count without (cpu, mode) (node_cpu_seconds_total{mode="idle"}) > 1` ## What this means here `NodeCPUPressureHigh` is PSI "some" for CPU: the fraction of wall-clock time during which at least one task was stalled waiting for a CPU. Under 5% is healthy. Averaged CPU% — what `kubectl top` shows — **hides this entirely**, which is why the alert uses PSI: it is the metric that surfaces the "spikes cause blips of delay" symptom operators actually notice. `NodeCPULoadHigh` is 5-minute load average divided by core count. Sustained above 1 means the run queue is longer than the number of CPUs. **Read them together.** CPU-PSI high *and* load high = genuine CPU contention. Load high *and* CPU-PSI normal = the queue is full of tasks blocked on **I/O**, not CPU — go to `/runbooks/node-io`, which on this cluster is the more dangerous of the two (etcd WAL and Longhorn replicas share one partition). Measured history: steady CPU PSI ran **15–23%** with load at or above core count, which is what drove the node resize from **6 to 8 vCPU**; afterwards PSI measured ~4.5–7.7%. So 0.15 is not an arbitrary threshold, it is the level the cluster actually lived at before the resize. ## Running the commands in this runbook `kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write `backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for agents here: every durable change goes through git. **`main` is a protected branch — a direct push is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, and it says so. ## First commands Distinguish CPU contention from I/O contention — do this first: ``` ssh k3s-prod-1 'cat /proc/pressure/cpu; echo; cat /proc/pressure/io; echo; uptime; nproc' ``` Who is burning it: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod top nodes; echo; kubectl --context k3s-prod top pods -A --sort-by=cpu | head -20' ``` ``` ssh k3s-prod-1 'top -b -n1 | head -20' ``` Are there VMs, and are any of them emulated (the worst case)? ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get vmi -A -o wide' ``` ``` ssh k3s-prod-1 'grep -c -E "vmx|svm" /proc/cpuinfo; ls -l /dev/kvm' ``` ## Likely causes, most likely first ### 1. It is actually I/O, not CPU Most common misread. A node blocked on disk has a long run queue and normal CPU-PSI. Confirm with `/proc/pressure/io` and go to `/runbooks/node-io`. On 2026-09-07 the node that died had I/O-PSI full at 11.36 while CPU looked unremarkable. ### 2. A KubeVirt VM running under TCG emulation `useEmulation: true` **silently forces TCG on KubeVirt v1.9** — it is set to `false` on purpose. A VM that lost `/dev/kvm` or was authored with emulation burns a whole core per vCPU. Heavy sustained TCG load has also degraded the Cilium datapath into intermittent LB-VIP unreachability (cleared by a restart), so an "intermittent networking" report alongside this alert is probably one cause, not two. Check `grep -c -E "vmx|svm" /proc/cpuinfo` (nested KVM is verified working; Proxmox nodes use `cpu_type=host`). ### 3. A Longhorn rebuild's checksumming Rebuilds are I/O-heavy but not free on CPU. Same remediation as `/runbooks/longhorn-degraded`: freeze with `concurrent-replica-rebuild-per-node-limit` `0`, thaw to `1`. ### 4. Prometheus rule evaluation or a Loki query `PersistentVolumeFillingUpPredicted` runs `predict_linear` over a 24h window across every PVC; a Grafana panel doing the same interactively is heavier still. Bursty, self-limiting, usually harmless. ### 5. Genuine capacity If nothing is anomalous the node is simply undersized again. The 6 → 8 vCPU resize is the precedent. A bpg cores change **reboots the VM**, so roll **one node at a time, waiting for Longhorn `robustness=healthy` between each**. ## When to escalate or stop - **If I/O-PSI is also elevated, stop treating this as a CPU problem** and switch to `/runbooks/node-io`. Wrong runbook is the main risk here. - Alone and steady, this alert is a capacity signal — schedule it, do not page on it. That is why it is a warning with a 15m `for`. - **Do not drain or cordon** to shed CPU load; this cluster does not drain. - Escalate to the operator for a vCPU change (it reboots the node) or to kill a VM you did not create. --- Source of truth: `runbooks/node-cpu.md`. Index: `/runbooks/` Records: `DECISIONS-LOG.md` (node vCPU 6→8 and the post-resize PSI measurements; KubeVirt `useEmulation` finding).