home-network-k3s-impl/runbooks/node-cpu.md

node-cpu — CPU pressure or run-queue depth high on a node

Alerts: NodeCPUPressureHigh (warning, 15m) and NodeCPULoadHigh (warning, 15m) File: platform/monitoring/alerts.yaml, group platform-health.nodes Expressions: rate(node_pressure_cpu_waiting_seconds_total[5m]) > 0.15 node_load5 / count without (cpu, mode) (node_cpu_seconds_total{mode="idle"}) > 1

What this means here

NodeCPUPressureHigh is PSI "some" for CPU: the fraction of wall-clock time during which at least one task was stalled waiting for a CPU. Under 5% is healthy. Averaged CPU% — what kubectl top shows — hides this entirely, which is why the alert uses PSI: it is the metric that surfaces the "spikes cause blips of delay" symptom operators actually notice.

NodeCPULoadHigh is 5-minute load average divided by core count. Sustained above 1 means the run queue is longer than the number of CPUs.

Read them together. CPU-PSI high and load high = genuine CPU contention. Load high and CPU-PSI normal = the queue is full of tasks blocked on I/O, not CPU — go to /runbooks/node-io, which on this cluster is the more dangerous of the two (etcd WAL and Longhorn replicas share one partition).

Measured history: steady CPU PSI ran 15–23% with load at or above core count, which is what drove the node resize from 6 to 8 vCPU; afterwards PSI measured ~4.5–7.7%. So 0.15 is not an arbitrary threshold, it is the level the cluster actually lived at before the resize.

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges both clusters into one kubeconfig and defaults its current-context to k3s-dev, so a command without --context silently answers about the wrong cluster. (CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... is equivalent.) Do not run two dev.sh invocations at once — they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git (git push origin main is the deploy). Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

Distinguish CPU contention from I/O contention — do this first:

ssh k3s-prod-1 'cat /proc/pressure/cpu; echo; cat /proc/pressure/io; echo; uptime; nproc'

Who is burning it:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod top nodes; echo; kubectl --context k3s-prod top pods -A --sort-by=cpu | head -20'

ssh k3s-prod-1 'top -b -n1 | head -20'

Are there VMs, and are any of them emulated (the worst case)?

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get vmi -A -o wide'

ssh k3s-prod-1 'grep -c -E "vmx|svm" /proc/cpuinfo; ls -l /dev/kvm'

Likely causes, most likely first

1. It is actually I/O, not CPU

Most common misread. A node blocked on disk has a long run queue and normal CPU-PSI. Confirm with /proc/pressure/io and go to /runbooks/node-io. On 2026-09-07 the node that died had I/O-PSI full at 11.36 while CPU looked unremarkable.

2. A KubeVirt VM running under TCG emulation

useEmulation: true silently forces TCG on KubeVirt v1.9 — it is set to false on purpose. A VM that lost /dev/kvm or was authored with emulation burns a whole core per vCPU. Heavy sustained TCG load has also degraded the Cilium datapath into intermittent LB-VIP unreachability (cleared by a restart), so an "intermittent networking" report alongside this alert is probably one cause, not two. Check grep -c -E "vmx|svm" /proc/cpuinfo (nested KVM is verified working; Proxmox nodes use cpu_type=host).

3. A Longhorn rebuild's checksumming

Rebuilds are I/O-heavy but not free on CPU. Same remediation as /runbooks/longhorn-degraded: freeze with concurrent-replica-rebuild-per-node-limit 0, thaw to 1.

4. Prometheus rule evaluation or a Loki query

PersistentVolumeFillingUpPredicted runs predict_linear over a 24h window across every PVC; a Grafana panel doing the same interactively is heavier still. Bursty, self-limiting, usually harmless.

5. Genuine capacity

If nothing is anomalous the node is simply undersized again. The 6 → 8 vCPU resize is the precedent. A bpg cores change reboots the VM, so roll one node at a time, waiting for Longhorn robustness=healthy between each.

When to escalate or stop

• If I/O-PSI is also elevated, stop treating this as a CPU problem and switch to /runbooks/node-io. Wrong runbook is the main risk here.
• Alone and steady, this alert is a capacity signal — schedule it, do not page on it. That is why it is a warning with a 15m for.
• Do not drain or cordon to shed CPU load; this cluster does not drain.
• Escalate to the operator for a vCPU change (it reboots the node) or to kill a VM you did not create.

---

Source of truth: runbooks/node-cpu.md. Index: /runbooks/ Records: DECISIONS-LOG.md (node vCPU 6→8 and the post-resize PSI measurements; KubeVirt useEmulation finding).