jakman85andClaude Opus 5
8e017fad91
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
166 lines
8.9 KiB
Markdown
166 lines
8.9 KiB
Markdown
# kubevirt-crashloop — A VM's virt-launcher pod is restarting repeatedly
|
|
|
|
**Alert:** `KubeVirtVMCrashLooping` (warning, 10m)
|
|
**File:** `platform/kubevirt/alerts.yaml`, group `platform-health.kubevirt`
|
|
**Expression:** `increase(kube_pod_container_status_restarts_total{pod=~"virt-launcher-.*"}[15m]) > 3`
|
|
|
|
## What this means here
|
|
|
|
More than three restarts of a `virt-launcher` pod in 15 minutes. The threshold is chosen to distinguish a
|
|
crash-**loop** from a one-off reboot: a guest that restarts once is not interesting, a guest that restarts
|
|
four times in a quarter of an hour is either panicking, being OOM-killed, or crashing qemu.
|
|
|
|
Note what layer this watches: the **pod**, not the guest. `virt-launcher` wraps qemu, so a guest kernel
|
|
panic, an OOM kill of the launcher, and a qemu crash all present the same way here. The distinction comes
|
|
from the logs and the events, not from the alert.
|
|
|
|
This is a `warning`, correctly: the VM tier is deliberately non-gating (`apps-vms` is `wait: false`) and a
|
|
crash-looping VM does not affect the container platform. The one caveat is that on this network the UniFi
|
|
VM is the home network's controller, so *that* VM crash-looping is user-visible.
|
|
|
|
Two platform facts that shape the likely causes:
|
|
|
|
- These are **HP EliteDesk 800 G5 Minis with ~22 GiB usable each**, running the whole platform. A VM is
|
|
the largest discretionary memory consumer on a node, so OOM is a live possibility rather than a
|
|
theoretical one.
|
|
- **`useEmulation: false` is set deliberately** — on KubeVirt v1.9, `true` silently forces TCG. A VM
|
|
running under TCG burns a whole core per vCPU, and heavy sustained TCG load has been observed to degrade
|
|
the Cilium datapath into intermittent LB-VIP unreachability (cleared by a restart). So "the VM is
|
|
unstable **and** the network is flaky" is usually one cause, not two.
|
|
## Running the commands in this runbook
|
|
|
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
|
that hang is not the cluster being down. Every command below runs as:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
|
|
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
|
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
|
|
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
|
|
the same thing next month as it does today; one that relies on the default does not, and the failure is
|
|
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
|
|
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
|
|
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
|
|
interfere.
|
|
|
|
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
|
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
|
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
|
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
|
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
|
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
|
|
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
|
and it says so.
|
|
|
|
## First commands
|
|
|
|
Which launcher, how many restarts, on which node:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get pods -A -l kubevirt.io=virt-launcher -o wide; echo; \
|
|
kubectl --context k3s-prod get vmi -A -o wide'
|
|
```
|
|
|
|
Why it died — the previous container's log is the one that matters:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n unifi logs -l kubevirt.io=virt-launcher --previous --tail=120 2>&1 | tail -60'
|
|
```
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n unifi describe pod -l kubevirt.io=virt-launcher 2>&1 | sed -n "/Last State/,\$p" | head -40'
|
|
```
|
|
|
|
Was it OOM-killed?
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get events -A --field-selector reason=OOMKilling --sort-by=.lastTimestamp | tail -10; echo; \
|
|
kubectl --context k3s-prod top pods -A --sort-by=memory | head -10'
|
|
```
|
|
|
|
```
|
|
ssh k3s-prod-1 'dmesg -T | grep -iE "oom|killed process" | tail -10; free -h; swapon --show'
|
|
```
|
|
|
|
(An empty `swapon --show` is what you want. On 2026-09-07 all three prod nodes came back from a reboot with
|
|
977 MiB of swap **active** while the role reported success — `SwapTotal == 0` is not evidence swap is off.
|
|
See `/runbooks/node-memory`.)
|
|
|
|
Is it really TCG rather than KVM?
|
|
|
|
```
|
|
ssh k3s-prod-1 'ls -l /dev/kvm; grep -c -E "vmx|svm" /proc/cpuinfo; ps -eo pid,pcpu,comm | grep -i qemu'
|
|
```
|
|
|
|
Is the disk healthy underneath?
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
|
|
-o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,STATE:.status.state | grep -v healthy'
|
|
```
|
|
|
|
## Likely causes, most likely first
|
|
|
|
### 1. The guest is panicking or the guest OS is broken
|
|
Read the serial console via the previous container's log. A guest that panics at boot after a change is
|
|
usually a bad cloud-init or a bad image. Two precise traps here: **KubeVirt caps inline `userData` at 2048
|
|
bytes** (the shipped payload is a deliberately minimal ~1.1 KB inline blob; `userDataSecretRef` was tried
|
|
and reverted), and **`apps-vms` must NOT carry `postBuild.substituteFrom`** — Flux's envsubst rewrote
|
|
`${...}` inside the cloud-init and corrupted it (commit `da1f833`). Do not restore substitution there for
|
|
consistency with the app waves.
|
|
|
|
### 2. The launcher is being OOM-killed
|
|
On ~22 GiB nodes running the whole platform, a VM with a generous memory request plus Prometheus (3Gi) and
|
|
Grafana (768Mi) is tight. Remediation: lower the VM's memory in git, or address node memory
|
|
(`/runbooks/node-memory`). Check swap is genuinely off first — active swap makes the kubelet's accounting
|
|
lie and eviction misbehave.
|
|
|
|
### 3. qemu is crashing
|
|
The launcher log will carry the qemu error. Most often a device or firmware mismatch: a UEFI image needs
|
|
`firmware.efi`, and `containerDisk` does not work here at all (it requires the ALPHA `ImageVolume` gate,
|
|
off on k3s — use a CDI DataVolume). See `/runbooks/kubevirt-launch-failed`.
|
|
|
|
### 4. The disk is unavailable or degraded mid-flight
|
|
A `longhorn-vm` volume that went degraded or faulted takes the guest with it. Cross-read
|
|
`/runbooks/longhorn-degraded` and `/runbooks/longhorn-faulted`. Remember `longhorn-vm` is **Block mode
|
|
only** and `migratable: true`; a VM on plain `longhorn` cannot live-migrate.
|
|
|
|
### 5. Node I/O pressure is starving the guest
|
|
A guest whose virtual disk cannot complete writes will wedge and be restarted. `/runbooks/node-io` — and
|
|
note the guest's writes land on the same partition as the embedded etcd WAL, so a thrashing VM is a
|
|
control-plane risk, not just its own problem.
|
|
|
|
### 6. It is running under TCG
|
|
`useEmulation: true` forces it on v1.9. Confirm with `/dev/kvm` presence and qemu CPU usage. Fix in git.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **Bound the damage first.** If the VM is thrashing the disk, its restarts are loading the node that
|
|
etcd writes to. If `/runbooks/node-io` is also firing, stopping the VM (an operator action on the `VM`
|
|
object, `spec.running: false` in git) is the right move — protecting the control plane beats keeping one
|
|
VM trying.
|
|
- **Do not delete the DataVolume or the PVC.** A re-import is hours of I/O on the shared partition.
|
|
- **Do not `kubectl drain`** the node to relocate it. This cluster does not drain, and a VM live-migrates
|
|
on drain **only** with per-VM `evictionStrategy: LiveMigrate`; a blanket setting would wedge drains for
|
|
RWO-disk VMs.
|
|
- Escalate if the guest needs in-guest repair — that is console work, and for the UniFi VM it is also
|
|
home-network-visible.
|
|
- If the alert is firing for a `virt-launcher` in a leaked test namespace, deleting that namespace is the
|
|
fix; leaked TCG test namespaces have caused API blips here before.
|
|
|
|
---
|
|
Source of truth: `runbooks/kubevirt-crashloop.md`. Index: `/runbooks/`
|
|
Records: `DECISIONS-LOG.md` Session 7 (`useEmulation`, `firmware.efi`, `containerDisk`/ImageVolume);
|
|
commit `da1f833` (cloud-init corrupted by envsubst); the 2026-09-07 swap-persistence finding.
|