jakman85andClaude Opus 5
8e017fad91
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
6.4 KiB
node-cpu — CPU pressure or run-queue depth high on a node
Alerts: NodeCPUPressureHigh (warning, 15m) and NodeCPULoadHigh (warning, 15m)
File: platform/monitoring/alerts.yaml, group platform-health.nodes
Expressions:
rate(node_pressure_cpu_waiting_seconds_total[5m]) > 0.15
node_load5 / count without (cpu, mode) (node_cpu_seconds_total{mode="idle"}) > 1
What this means here
NodeCPUPressureHigh is PSI "some" for CPU: the fraction of wall-clock time during which at least
one task was stalled waiting for a CPU. Under 5% is healthy. Averaged CPU% — what kubectl top
shows — hides this entirely, which is why the alert uses PSI: it is the metric that surfaces the
"spikes cause blips of delay" symptom operators actually notice.
NodeCPULoadHigh is 5-minute load average divided by core count. Sustained above 1 means the run
queue is longer than the number of CPUs.
Read them together. CPU-PSI high and load high = genuine CPU contention. Load high and CPU-PSI
normal = the queue is full of tasks blocked on I/O, not CPU — go to /runbooks/node-io, which on
this cluster is the more dangerous of the two (etcd WAL and Longhorn replicas share one partition).
Measured history: steady CPU PSI ran 15–23% with load at or above core count, which is what drove the node resize from 6 to 8 vCPU; afterwards PSI measured ~4.5–7.7%. So 0.15 is not an arbitrary threshold, it is the level the cluster actually lived at before the resize.
Running the commands in this runbook
kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's k3s-oidc context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and that default has already changed once — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two dev.sh invocations at once; they share one container and
interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup:
longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to
Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io,
settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.
Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for
agents here: every durable change goes through git. main is a protected branch — a direct push
is rejected. A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.
First commands
Distinguish CPU contention from I/O contention — do this first:
ssh k3s-prod-1 'cat /proc/pressure/cpu; echo; cat /proc/pressure/io; echo; uptime; nproc'
Who is burning it:
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod top nodes; echo; kubectl --context k3s-prod top pods -A --sort-by=cpu | head -20'
ssh k3s-prod-1 'top -b -n1 | head -20'
Are there VMs, and are any of them emulated (the worst case)?
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get vmi -A -o wide'
ssh k3s-prod-1 'grep -c -E "vmx|svm" /proc/cpuinfo; ls -l /dev/kvm'
Likely causes, most likely first
1. It is actually I/O, not CPU
Most common misread. A node blocked on disk has a long run queue and normal CPU-PSI. Confirm with
/proc/pressure/io and go to /runbooks/node-io. On 2026-09-07 the node that died had I/O-PSI full
at 11.36 while CPU looked unremarkable.
2. A KubeVirt VM running under TCG emulation
useEmulation: true silently forces TCG on KubeVirt v1.9 — it is set to false on purpose. A VM
that lost /dev/kvm or was authored with emulation burns a whole core per vCPU. Heavy sustained TCG
load has also degraded the Cilium datapath into intermittent LB-VIP unreachability (cleared by a
restart), so an "intermittent networking" report alongside this alert is probably one cause, not two.
Check grep -c -E "vmx|svm" /proc/cpuinfo (nested KVM is verified working; Proxmox nodes use
cpu_type=host).
3. A Longhorn rebuild's checksumming
Rebuilds are I/O-heavy but not free on CPU. Same remediation as /runbooks/longhorn-degraded: freeze
with concurrent-replica-rebuild-per-node-limit 0, thaw to 1.
4. Prometheus rule evaluation or a Loki query
PersistentVolumeFillingUpPredicted runs predict_linear over a 24h window across every PVC; a
Grafana panel doing the same interactively is heavier still. Bursty, self-limiting, usually harmless.
5. Genuine capacity
If nothing is anomalous the node is simply undersized again. The 6 → 8 vCPU resize is the precedent.
A bpg cores change reboots the VM, so roll one node at a time, waiting for Longhorn
robustness=healthy between each.
When to escalate or stop
- If I/O-PSI is also elevated, stop treating this as a CPU problem and switch to
/runbooks/node-io. Wrong runbook is the main risk here. - Alone and steady, this alert is a capacity signal — schedule it, do not page on it. That is why it
is a warning with a 15m
for. - Do not drain or cordon to shed CPU load; this cluster does not drain.
- Escalate to the operator for a vCPU change (it reboots the node) or to kill a VM you did not create.
Source of truth: runbooks/node-cpu.md. Index: /runbooks/
Records: DECISIONS-LOG.md (node vCPU 6→8 and the post-resize PSI measurements; KubeVirt
useEmulation finding).