jakman85andClaude Opus 5
8e017fad91
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
176 lines
9.4 KiB
Markdown
176 lines
9.4 KiB
Markdown
# kubevirt-launch-failed — One or more VirtualMachineInstances are in the Failed phase
|
|
|
|
**Alert:** `KubeVirtVMLaunchFailing` (critical, 5m)
|
|
**File:** `platform/kubevirt/alerts.yaml`, group `platform-health.kubevirt`
|
|
**Expression:** `sum by (namespace) (kubevirt_vmi_phase_count{phase="Failed"}) > 0`
|
|
|
|
## What this means here
|
|
|
|
A VMI reached the `Failed` phase — a scheduling, launch or qemu failure. The IaaS path is broken for that
|
|
workload, which is why it is page-worthy even though the VM tier is deliberately **non-gating**
|
|
(`apps-vms` is `wait: false`, `dependsOn: platform-config`, because a VM's image import and in-guest
|
|
install are legitimately long-not-Ready).
|
|
|
|
What actually runs here: the `unifi` namespace holds the UniFi VM (rootdisk 40 GiB, Block mode on
|
|
`longhorn-vm`). **UniFi OS Server is not yet installed in the guest** — that is known, documented state,
|
|
not a launch failure.
|
|
|
|
**Live note, measured 2026-09-13:** `kubevirt_vmi_phase_count` returned **0 series** on prod. That metric
|
|
only exists while VMIs do, so this alert is currently evaluating against nothing. Confirm the real state
|
|
from `kubectl get vmi -A`, not from this alert's silence. (`kubevirt-prometheus-metrics` *is* a live scrape
|
|
job and `kubevirt_virt_api_ready_status` does publish, so the target is up — see
|
|
`/runbooks/kubevirt-control-plane`.)
|
|
|
|
### The three k3s-specific launch failures this platform already hit and fixed
|
|
|
|
Recorded from bring-up, and worth checking against any new VM definition because all three were
|
|
non-obvious:
|
|
|
|
1. **`containerDisk` requires the ALPHA `ImageVolume` feature gate**, which is off on k3s. The fix was to
|
|
switch to a **CDI DataVolume on Longhorn** instead. A VM authored with `containerDisk` will not launch
|
|
here.
|
|
2. **A UEFI image needs `firmware.efi`** set on the VM. Without it, boot fails.
|
|
3. **`useEmulation: true` silently forces TCG on KubeVirt v1.9** — it is set to `false` deliberately. A VM
|
|
that appears to launch but burns a whole core per vCPU is this; it also degrades the Cilium datapath
|
|
under sustained load.
|
|
|
|
Also recorded: a `virt-handler` certificate error seen during that work was **unrelated and cosmetic** —
|
|
do not chase it.
|
|
## Running the commands in this runbook
|
|
|
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
|
that hang is not the cluster being down. Every command below runs as:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
|
|
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
|
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
|
|
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
|
|
the same thing next month as it does today; one that relies on the default does not, and the failure is
|
|
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
|
|
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
|
|
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
|
|
interfere.
|
|
|
|
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
|
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
|
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
|
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
|
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
|
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
|
|
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
|
and it says so.
|
|
|
|
## First commands
|
|
|
|
Which VMI, in which namespace, and what phase:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get vmi -A -o wide; echo; \
|
|
kubectl --context k3s-prod get vm -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,STATUS:.status.printableStatus,READY:.status.ready'
|
|
```
|
|
|
|
The reason — the VMI status conditions and the launcher pod's events are where it is stated:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n unifi describe vmi unifi 2>&1 | sed -n "/Status:/,\$p" | head -60'
|
|
```
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n unifi get pods -l kubevirt.io=virt-launcher -o wide; echo; \
|
|
kubectl --context k3s-prod -n unifi describe pod -l kubevirt.io=virt-launcher 2>&1 | sed -n "/Events:/,\$p"'
|
|
```
|
|
|
|
Is the disk there? (A missing or still-importing DataVolume is the most common cause.)
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get datavolumes.cdi.kubevirt.io -A \
|
|
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PHASE:.status.phase,PROGRESS:.status.progress; echo; \
|
|
kubectl --context k3s-prod get pvc -A | grep -Ei "rootdisk|dv|unifi"'
|
|
```
|
|
|
|
Is nested virtualisation actually available on the node it was scheduled to?
|
|
|
|
```
|
|
ssh k3s-prod-1 'ls -l /dev/kvm; grep -c -E "vmx|svm" /proc/cpuinfo'
|
|
```
|
|
|
|
Is the storage class right for a live-migratable VM?
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get sc; echo; \
|
|
kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy'
|
|
```
|
|
|
|
And whether the KubeVirt control plane can act at all:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kubevirt get pods'
|
|
```
|
|
|
|
## Likely causes, most likely first
|
|
|
|
### 1. The disk is not ready — a CDI import that has not completed or has stalled
|
|
The VM cannot start without its DataVolume. Go to `/runbooks/cdi-import-stuck`. The specific recorded
|
|
failure: the **3-replica default class failed the UniFi 60Gi import with insufficient storage /
|
|
`ReplicaSchedulingFailure`**, which is why a dedicated single-replica **`longhorn-scratch`** class exists
|
|
and is referenced by the CDI CR's `scratchSpaceStorageClass`.
|
|
|
|
### 2. The VM is authored in a way k3s does not support
|
|
The three items above: `containerDisk` without the ALPHA `ImageVolume` gate (use a CDI DataVolume),
|
|
missing `firmware.efi` for a UEFI image, or `useEmulation: true` forcing TCG. All are git changes to the
|
|
VM definition.
|
|
|
|
### 3. The cloud-init payload is too large or corrupted
|
|
Two precise traps. **KubeVirt caps inline `userData` at 2048 bytes** — the original rich cloud-init
|
|
exceeded it; `userDataSecretRef` was tried and reverted, and the shipped state is a minimal ~1.1 KB inline
|
|
payload. And: **`apps-vms` must NOT carry `postBuild.substituteFrom`** — Flux's envsubst rewrote `${...}`
|
|
inside the cloud-init and corrupted it (commit `da1f833`). *Do not restore substitution there for
|
|
consistency with the app waves.* If a VM stopped booting right after someone "tidied up" the Flux
|
|
Kustomizations, this is it.
|
|
|
|
### 4. It cannot be scheduled
|
|
No node with room, or a storage constraint. A live-migratable VM needs **`longhorn-vm`**: RWX,
|
|
**Block mode only** (migratable RWX is a raw block device, not the share-manager/NFS path), annotated
|
|
`storageclass.kubevirt.io/is-default-virt-class: "true"` — which is *not* the cluster default class.
|
|
A VM pointed at plain `longhorn` will not live-migrate.
|
|
|
|
### 5. The CDI importer cannot open the raw Block device
|
|
`nonroot-devices: true` is set on the k3s servers because the non-root CDI importer must open a raw Block
|
|
device that Kubernetes never chowns for it (commit `df5979f`). A permission error in the importer on a
|
|
Block PVC is this.
|
|
|
|
### 6. The KubeVirt control plane is down
|
|
Then nothing can launch. `/runbooks/kubevirt-control-plane`.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **Scope it before escalating.** The VM tier is non-gating and does not affect the container platform.
|
|
One failed VM is not a platform outage — but the UniFi VM is the home network's controller, so treat
|
|
that one as user-visible.
|
|
- **Do not delete the VM or its DataVolume to "start clean".** A 60 GiB re-import is hours of I/O on disks
|
|
that share a partition with the embedded etcd WAL, and that is the mechanism behind the 2026-09-07 node
|
|
outage. Fix the definition, not the data.
|
|
- **Do not set `evictionStrategy: LiveMigrate` cluster-wide.** A blanket setting would **wedge drains**
|
|
for RWO-disk VMs; it is per-VM on purpose.
|
|
- **Do not add `postBuild.substituteFrom` to `apps-vms`.** It corrupts cloud-init.
|
|
- Escalate for anything that needs a fresh image import during a storage incident, and for changes to the
|
|
k3s server flags (`nonroot-devices`) — those are Ansible/operator work.
|
|
|
|
---
|
|
Source of truth: `runbooks/kubevirt-launch-failed.md`. Index: `/runbooks/`
|
|
Records: `DECISIONS-LOG.md` Session 7 (the three k3s-specific launch failures); commits `da1f833`
|
|
(cloud-init corrupted by envsubst) and `df5979f` (`nonroot-devices`); the `longhorn-scratch` /
|
|
UniFi 60Gi import finding.
|