jakman85andClaude Opus 5
b7341b329b
Three corrections after rebasing onto main. 1. THE GUARD CAUGHT A REAL ONE ON ITS FIRST CONTACT WITH main. main had gained `CNPGPrimariesCoLocated` (platform/cnpg/alerts.yaml) pointing at /runbooks/cnpg-primary-spread, with no runbook. The check failed the rebase exactly as designed — 46 annotations, 35 names, 34 present — naming both the missing file AND the missing configMapGenerator entry. runbooks/cnpg-primary-spread.md now exists: what co-located primaries actually cost here (blast radius, not an outage, because instances: 3 + required anti-affinity), why the drift is uncorrectable-by-declaration (CNPG has no primary-placement field), the readiness gate that keeps the rule quiet during a real node outage, and the ~9s-per-database switchover rebalance, one cluster at a time. The 2026-09-07 outage in which all three primaries sat on one node is the reason the alert exists at all. 2. WHOLE-REPO SCOPE, NOT A HAND-WRITTEN STAGE. main's refactor replaced ci/l0.sh's per-check "UNCONDITIONAL" stages with a declaration: `conformance.home.k3s/scope: "whole-repo"`, unioned into every gate run by Stage 3b's `--include-scope`. My +38-line Stage 3f is removed (ci/l0.sh is now byte-identical to main) and the annotation is declared instead. Assertion (7) of the check was rewritten to match: it no longer asserts its own path appears in ci/l0.sh — it asserts the test DECLARES the scope and that ci/l0.sh still passes --include-scope, since neither half alone makes the check load-bearing. That matters here because ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02", not R-OBS-04, and maps platform/kubevirt/* and runbooks/* to nothing at all. 3. A FACTUAL CORRECTION IN ALL 35 RUNBOOKS. Every page claimed `dev.sh` defaults its current-context to k3s-dev. main's `9b4744d fix(devcontainer): dev.sh defaults to prod` made that false, and a runbook that misstates which cluster a command talks to is worse than one that says nothing. The pages now say what is durably true: the default has already changed once, so naming `--context k3s-prod` is what makes a command mean the same thing next month, and the per-invocation banner is there to be read. The check's own failure message was corrected the same way. Re-verified after all three: check green at 374 assertions (46 annotations, 35/35 runbooks present, 35/35 projected), re-applied to prod, and re-probed in-cluster through the Gateway — node-down, keycloak-down, keycloak-client-secrets, cnpg-archiving, longhorn-faulted, vpn-down, node-io and cnpg-primary-spread all HTTP 200 with text/plain; charset=utf-8 and 8.9-12.1 KB of real body; /runbooks/ index 200; an unknown name 404s with a pointer rather than a blank page; and homepage still answers the apex `/`, confirming Gateway API longest-path precedence. Full merge gate: PASS (spec-immutability, lint, l0, render-golden, l1 all green; prom-rules SKIPPED for want of promtool in the image; L2/L3 SKIPPED, no cluster). Co-Authored-By: Claude Opus 5 <[email protected]>
6.3 KiB
node-cpu — CPU pressure or run-queue depth high on a node
Alerts: NodeCPUPressureHigh (warning, 15m) and NodeCPULoadHigh (warning, 15m)
File: platform/monitoring/alerts.yaml, group platform-health.nodes
Expressions:
rate(node_pressure_cpu_waiting_seconds_total[5m]) > 0.15
node_load5 / count without (cpu, mode) (node_cpu_seconds_total{mode="idle"}) > 1
What this means here
NodeCPUPressureHigh is PSI "some" for CPU: the fraction of wall-clock time during which at least
one task was stalled waiting for a CPU. Under 5% is healthy. Averaged CPU% — what kubectl top
shows — hides this entirely, which is why the alert uses PSI: it is the metric that surfaces the
"spikes cause blips of delay" symptom operators actually notice.
NodeCPULoadHigh is 5-minute load average divided by core count. Sustained above 1 means the run
queue is longer than the number of CPUs.
Read them together. CPU-PSI high and load high = genuine CPU contention. Load high and CPU-PSI
normal = the queue is full of tasks blocked on I/O, not CPU — go to /runbooks/node-io, which on
this cluster is the more dangerous of the two (etcd WAL and Longhorn replicas share one partition).
Measured history: steady CPU PSI ran 15–23% with load at or above core count, which is what drove the node resize from 6 to 8 vCPU; afterwards PSI measured ~4.5–7.7%. So 0.15 is not an arbitrary threshold, it is the level the cluster actually lived at before the resize.
Running the commands in this runbook
kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's k3s-oidc context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and that default has already changed once — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two dev.sh invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup:
longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to
Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io,
settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.
Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for
agents here: every durable change goes through git (git push origin main is the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
First commands
Distinguish CPU contention from I/O contention — do this first:
ssh k3s-prod-1 'cat /proc/pressure/cpu; echo; cat /proc/pressure/io; echo; uptime; nproc'
Who is burning it:
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod top nodes; echo; kubectl --context k3s-prod top pods -A --sort-by=cpu | head -20'
ssh k3s-prod-1 'top -b -n1 | head -20'
Are there VMs, and are any of them emulated (the worst case)?
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get vmi -A -o wide'
ssh k3s-prod-1 'grep -c -E "vmx|svm" /proc/cpuinfo; ls -l /dev/kvm'
Likely causes, most likely first
1. It is actually I/O, not CPU
Most common misread. A node blocked on disk has a long run queue and normal CPU-PSI. Confirm with
/proc/pressure/io and go to /runbooks/node-io. On 2026-09-07 the node that died had I/O-PSI full
at 11.36 while CPU looked unremarkable.
2. A KubeVirt VM running under TCG emulation
useEmulation: true silently forces TCG on KubeVirt v1.9 — it is set to false on purpose. A VM
that lost /dev/kvm or was authored with emulation burns a whole core per vCPU. Heavy sustained TCG
load has also degraded the Cilium datapath into intermittent LB-VIP unreachability (cleared by a
restart), so an "intermittent networking" report alongside this alert is probably one cause, not two.
Check grep -c -E "vmx|svm" /proc/cpuinfo (nested KVM is verified working; Proxmox nodes use
cpu_type=host).
3. A Longhorn rebuild's checksumming
Rebuilds are I/O-heavy but not free on CPU. Same remediation as /runbooks/longhorn-degraded: freeze
with concurrent-replica-rebuild-per-node-limit 0, thaw to 1.
4. Prometheus rule evaluation or a Loki query
PersistentVolumeFillingUpPredicted runs predict_linear over a 24h window across every PVC; a
Grafana panel doing the same interactively is heavier still. Bursty, self-limiting, usually harmless.
5. Genuine capacity
If nothing is anomalous the node is simply undersized again. The 6 → 8 vCPU resize is the precedent.
A bpg cores change reboots the VM, so roll one node at a time, waiting for Longhorn
robustness=healthy between each.
When to escalate or stop
- If I/O-PSI is also elevated, stop treating this as a CPU problem and switch to
/runbooks/node-io. Wrong runbook is the main risk here. - Alone and steady, this alert is a capacity signal — schedule it, do not page on it. That is why it
is a warning with a 15m
for. - Do not drain or cordon to shed CPU load; this cluster does not drain.
- Escalate to the operator for a vCPU change (it reboots the node) or to kill a VM you did not create.
Source of truth: runbooks/node-cpu.md. Index: /runbooks/
Records: DECISIONS-LOG.md (node vCPU 6→8 and the post-resize PSI measurements; KubeVirt
useEmulation finding).