jakman85andClaude Opus 5
b7341b329b
Three corrections after rebasing onto main. 1. THE GUARD CAUGHT A REAL ONE ON ITS FIRST CONTACT WITH main. main had gained `CNPGPrimariesCoLocated` (platform/cnpg/alerts.yaml) pointing at /runbooks/cnpg-primary-spread, with no runbook. The check failed the rebase exactly as designed — 46 annotations, 35 names, 34 present — naming both the missing file AND the missing configMapGenerator entry. runbooks/cnpg-primary-spread.md now exists: what co-located primaries actually cost here (blast radius, not an outage, because instances: 3 + required anti-affinity), why the drift is uncorrectable-by-declaration (CNPG has no primary-placement field), the readiness gate that keeps the rule quiet during a real node outage, and the ~9s-per-database switchover rebalance, one cluster at a time. The 2026-09-07 outage in which all three primaries sat on one node is the reason the alert exists at all. 2. WHOLE-REPO SCOPE, NOT A HAND-WRITTEN STAGE. main's refactor replaced ci/l0.sh's per-check "UNCONDITIONAL" stages with a declaration: `conformance.home.k3s/scope: "whole-repo"`, unioned into every gate run by Stage 3b's `--include-scope`. My +38-line Stage 3f is removed (ci/l0.sh is now byte-identical to main) and the annotation is declared instead. Assertion (7) of the check was rewritten to match: it no longer asserts its own path appears in ci/l0.sh — it asserts the test DECLARES the scope and that ci/l0.sh still passes --include-scope, since neither half alone makes the check load-bearing. That matters here because ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02", not R-OBS-04, and maps platform/kubevirt/* and runbooks/* to nothing at all. 3. A FACTUAL CORRECTION IN ALL 35 RUNBOOKS. Every page claimed `dev.sh` defaults its current-context to k3s-dev. main's `9b4744d fix(devcontainer): dev.sh defaults to prod` made that false, and a runbook that misstates which cluster a command talks to is worse than one that says nothing. The pages now say what is durably true: the default has already changed once, so naming `--context k3s-prod` is what makes a command mean the same thing next month, and the per-invocation banner is there to be read. The check's own failure message was corrected the same way. Re-verified after all three: check green at 374 assertions (46 annotations, 35/35 runbooks present, 35/35 projected), re-applied to prod, and re-probed in-cluster through the Gateway — node-down, keycloak-down, keycloak-client-secrets, cnpg-archiving, longhorn-faulted, vpn-down, node-io and cnpg-primary-spread all HTTP 200 with text/plain; charset=utf-8 and 8.9-12.1 KB of real body; /runbooks/ index 200; an unknown name 404s with a pointer rather than a blank page; and homepage still answers the apex `/`, confirming Gateway API longest-path precedence. Full merge gate: PASS (spec-immutability, lint, l0, render-golden, l1 all green; prom-rules SKIPPED for want of promtool in the image; L2/L3 SKIPPED, no cluster). Co-Authored-By: Claude Opus 5 <[email protected]>
126 lines
6.3 KiB
Markdown
126 lines
6.3 KiB
Markdown
# node-cpu — CPU pressure or run-queue depth high on a node
|
||
|
||
**Alerts:** `NodeCPUPressureHigh` (warning, 15m) and `NodeCPULoadHigh` (warning, 15m)
|
||
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes`
|
||
**Expressions:**
|
||
`rate(node_pressure_cpu_waiting_seconds_total[5m]) > 0.15`
|
||
`node_load5 / count without (cpu, mode) (node_cpu_seconds_total{mode="idle"}) > 1`
|
||
|
||
## What this means here
|
||
|
||
`NodeCPUPressureHigh` is PSI "some" for CPU: the fraction of wall-clock time during which at least
|
||
one task was stalled waiting for a CPU. Under 5% is healthy. Averaged CPU% — what `kubectl top`
|
||
shows — **hides this entirely**, which is why the alert uses PSI: it is the metric that surfaces the
|
||
"spikes cause blips of delay" symptom operators actually notice.
|
||
|
||
`NodeCPULoadHigh` is 5-minute load average divided by core count. Sustained above 1 means the run
|
||
queue is longer than the number of CPUs.
|
||
|
||
**Read them together.** CPU-PSI high *and* load high = genuine CPU contention. Load high *and* CPU-PSI
|
||
normal = the queue is full of tasks blocked on **I/O**, not CPU — go to `/runbooks/node-io`, which on
|
||
this cluster is the more dangerous of the two (etcd WAL and Longhorn replicas share one partition).
|
||
|
||
Measured history: steady CPU PSI ran **15–23%** with load at or above core count, which is what drove
|
||
the node resize from **6 to 8 vCPU**; afterwards PSI measured ~4.5–7.7%. So 0.15 is not an arbitrary
|
||
threshold, it is the level the cluster actually lived at before the resize.
|
||
## Running the commands in this runbook
|
||
|
||
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
||
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
||
that hang is not the cluster being down. Every command below runs as:
|
||
|
||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
||
|
||
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
||
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
|
||
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
|
||
the same thing next month as it does today; one that relies on the default does not, and the failure is
|
||
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
|
||
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
|
||
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
|
||
interfere.
|
||
|
||
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
||
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
||
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
||
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
||
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
||
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||
|
||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||
|
||
## First commands
|
||
|
||
Distinguish CPU contention from I/O contention — do this first:
|
||
|
||
```
|
||
ssh k3s-prod-1 'cat /proc/pressure/cpu; echo; cat /proc/pressure/io; echo; uptime; nproc'
|
||
```
|
||
|
||
Who is burning it:
|
||
|
||
```
|
||
infra/devcontainer/dev.sh run bash -c \
|
||
'kubectl --context k3s-prod top nodes; echo; kubectl --context k3s-prod top pods -A --sort-by=cpu | head -20'
|
||
```
|
||
|
||
```
|
||
ssh k3s-prod-1 'top -b -n1 | head -20'
|
||
```
|
||
|
||
Are there VMs, and are any of them emulated (the worst case)?
|
||
|
||
```
|
||
infra/devcontainer/dev.sh run bash -c \
|
||
'kubectl --context k3s-prod get vmi -A -o wide'
|
||
```
|
||
|
||
```
|
||
ssh k3s-prod-1 'grep -c -E "vmx|svm" /proc/cpuinfo; ls -l /dev/kvm'
|
||
```
|
||
|
||
## Likely causes, most likely first
|
||
|
||
### 1. It is actually I/O, not CPU
|
||
Most common misread. A node blocked on disk has a long run queue and normal CPU-PSI. Confirm with
|
||
`/proc/pressure/io` and go to `/runbooks/node-io`. On 2026-09-07 the node that died had I/O-PSI full
|
||
at 11.36 while CPU looked unremarkable.
|
||
|
||
### 2. A KubeVirt VM running under TCG emulation
|
||
`useEmulation: true` **silently forces TCG on KubeVirt v1.9** — it is set to `false` on purpose. A VM
|
||
that lost `/dev/kvm` or was authored with emulation burns a whole core per vCPU. Heavy sustained TCG
|
||
load has also degraded the Cilium datapath into intermittent LB-VIP unreachability (cleared by a
|
||
restart), so an "intermittent networking" report alongside this alert is probably one cause, not two.
|
||
Check `grep -c -E "vmx|svm" /proc/cpuinfo` (nested KVM is verified working; Proxmox nodes use
|
||
`cpu_type=host`).
|
||
|
||
### 3. A Longhorn rebuild's checksumming
|
||
Rebuilds are I/O-heavy but not free on CPU. Same remediation as `/runbooks/longhorn-degraded`: freeze
|
||
with `concurrent-replica-rebuild-per-node-limit` `0`, thaw to `1`.
|
||
|
||
### 4. Prometheus rule evaluation or a Loki query
|
||
`PersistentVolumeFillingUpPredicted` runs `predict_linear` over a 24h window across every PVC; a
|
||
Grafana panel doing the same interactively is heavier still. Bursty, self-limiting, usually harmless.
|
||
|
||
### 5. Genuine capacity
|
||
If nothing is anomalous the node is simply undersized again. The 6 → 8 vCPU resize is the precedent.
|
||
A bpg cores change **reboots the VM**, so roll **one node at a time, waiting for Longhorn
|
||
`robustness=healthy` between each**.
|
||
|
||
## When to escalate or stop
|
||
|
||
- **If I/O-PSI is also elevated, stop treating this as a CPU problem** and switch to
|
||
`/runbooks/node-io`. Wrong runbook is the main risk here.
|
||
- Alone and steady, this alert is a capacity signal — schedule it, do not page on it. That is why it
|
||
is a warning with a 15m `for`.
|
||
- **Do not drain or cordon** to shed CPU load; this cluster does not drain.
|
||
- Escalate to the operator for a vCPU change (it reboots the node) or to kill a VM you did not create.
|
||
|
||
---
|
||
Source of truth: `runbooks/node-cpu.md`. Index: `/runbooks/`
|
||
Records: `DECISIONS-LOG.md` (node vCPU 6→8 and the post-resize PSI measurements; KubeVirt
|
||
`useEmulation` finding).
|