jakman85andClaude Opus 5
8e017fad91
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
159 lines
8.8 KiB
Markdown
159 lines
8.8 KiB
Markdown
# kubevirt-control-plane — The KubeVirt control plane is degraded
|
|
|
|
**Alert:** `KubeVirtControlPlaneDown` (critical, 10m)
|
|
**File:** `platform/kubevirt/alerts.yaml`, group `platform-health.kubevirt`
|
|
**Expression:**
|
|
```
|
|
(sum(kubevirt_virt_controller_ready_status) == 0)
|
|
or (sum(kubevirt_virt_api_ready_status) == 0)
|
|
or (absent(up{job="kubevirt-prometheus-metrics"} == 1))
|
|
```
|
|
|
|
## What this means here
|
|
|
|
No `virt-controller` or no `virt-api` replica reports ready, so **VM lifecycle operations — create, start,
|
|
stop, migrate — will fail** (docs/06). Note carefully what this does *not* mean: **already-running VMs
|
|
keep running.** `virt-launcher` pods and their qemu processes are independent of the control plane. So
|
|
this is an "I cannot manage VMs" alert, not "the VMs are down".
|
|
|
|
The `absent()` clause also fires if the whole KubeVirt metrics target disappears. The scrape job label is
|
|
**`kubevirt-prometheus-metrics`** (the Service name), **not** `kubevirt` — that precision is a fix: this
|
|
very alert once used a non-existent metric with the wrong job label and fired falsely (commit `b57b3e9`).
|
|
There is a sibling precedent too: a broken HTTP CDI ServiceMonitor caused a false `CDIOperatorDown`
|
|
(commit `6a7e9d1`). **So the first question with this alert is always whether it is telling the truth.**
|
|
|
|
Live state, measured 2026-09-13: `kubevirt-prometheus-metrics` **is** among Prometheus's 19 scrape jobs
|
|
and `kubevirt_virt_api_ready_status` returns **2 series**, so the first two clauses are evaluable and
|
|
quiet. (`kubevirt_vmi_phase_count` was absent at the same moment — that metric only exists when VMIs do,
|
|
which affects `/runbooks/kubevirt-launch-failed`, not this rule.)
|
|
|
|
Platform context: **KubeVirt v1.9.0 + CDI v1.66.0**, on real nested KVM (`/dev/kvm` present, vmx/svm
|
|
verified; the Proxmox nodes use `cpu_type=host`). The `kubevirt` and `cdi` namespaces deliberately carry
|
|
**no `app-namespace` label**, so Kyverno does not generate the default-deny baseline into them — it would
|
|
sever the virt control plane. Do not "fix" that by adding the label.
|
|
## Running the commands in this runbook
|
|
|
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
|
that hang is not the cluster being down. Every command below runs as:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
|
|
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
|
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
|
|
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
|
|
the same thing next month as it does today; one that relies on the default does not, and the failure is
|
|
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
|
|
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
|
|
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
|
|
interfere.
|
|
|
|
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
|
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
|
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
|
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
|
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
|
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
|
|
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
|
and it says so.
|
|
|
|
## First commands
|
|
|
|
Is the control plane actually unhealthy, or is this a metrics artefact?
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n kubevirt get pods -o wide; echo; \
|
|
kubectl --context k3s-prod get kubevirts.kubevirt.io -A \
|
|
-o custom-columns=NAME:.metadata.name,PHASE:.status.phase,READY:.status.conditions[?\(@.type==\"Available\"\)].status'
|
|
```
|
|
|
|
Are the VMs themselves still running? (This is the reassuring half.)
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get vmi -A -o wide; echo; \
|
|
kubectl --context k3s-prod get vm -A'
|
|
```
|
|
|
|
Is the scrape target up, and do the two metrics exist?
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
|
|
for q in "up{job=\"kubevirt-prometheus-metrics\"}" "sum(kubevirt_virt_controller_ready_status)" "sum(kubevirt_virt_api_ready_status)"; do \
|
|
printf "%-55s " "$q"; curl -s -G --data-urlencode "query=$q" localhost:19090/api/v1/query \
|
|
| python3 -c "import json,sys;print([x[\"value\"][1] for x in json.load(sys.stdin)[\"data\"][\"result\"]] or \"ABSENT\")"; done'
|
|
```
|
|
|
|
Why a component is unready:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n kubevirt logs deploy/virt-controller --tail=80 2>&1 | tail -30; echo ====; \
|
|
kubectl --context k3s-prod -n kubevirt logs deploy/virt-api --tail=80 2>&1 | tail -30; echo ====; \
|
|
kubectl --context k3s-prod -n kubevirt get events --sort-by=.lastTimestamp | tail -20'
|
|
```
|
|
|
|
Is the Flux tier that owns KubeVirt healthy?
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \
|
|
-o custom-columns=NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message'
|
|
```
|
|
|
|
## Likely causes, most likely first
|
|
|
|
### 1. The metrics target, not the control plane
|
|
Check the pods first. If `virt-api` and `virt-controller` are Running and Ready, this is a monitoring
|
|
fault — and there is direct precedent for false positives on this exact alert (`b57b3e9`) and its CDI
|
|
sibling (`6a7e9d1`). Remediation: fix the ServiceMonitor/metric in git. Cross-read
|
|
`/runbooks/targets-down`.
|
|
|
|
### 2. A node that hosted the control-plane pods went away
|
|
`virt-controller` and `virt-api` are ordinary Deployments. Go to `/runbooks/node-down`.
|
|
|
|
### 3. The KubeVirt Flux tier failed to reconcile
|
|
Specific known hazard: **`platform/kubevirt` still uses remote kustomize bases**, and the
|
|
kustomize-controller's git fetch has a **27s timeout** that remote GitHub bases flake against — a
|
|
`BuildFail` there is probably this. `external-snapshotter` was vendored into the repo for exactly this
|
|
reason; vendoring KubeVirt the same way is the durable fix. See `/runbooks/flux-reconcile`.
|
|
|
|
### 4. Node pressure
|
|
Heavy sustained node load — TCG-emulated VMs are the recorded example — can degrade the Cilium datapath
|
|
into intermittent LB-VIP reachability problems, which looks like control-plane flapping. A restart
|
|
cleared it. Cross-read `/runbooks/node-io` and `/runbooks/node-cpu`.
|
|
|
|
### 5. A bad operator upgrade
|
|
`/runbooks/update-rollback`.
|
|
|
|
### 6. A NetworkPolicy severed the control plane
|
|
Only if someone added `homelab.keithjackson.dev/app-namespace=true` to `kubevirt` or `cdi`. Those
|
|
namespaces are deliberately unlabelled so Kyverno does not inject a default-deny baseline that would cut
|
|
virt-handler off. Reverting that label is the fix.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **Prioritise honestly.** VM *lifecycle* is broken; running VMs are not. The UniFi VM keeps serving.
|
|
Treat this as urgent-but-not-catastrophic unless a VM also needs to start.
|
|
- **Do not drain a node to move virt pods.** This cluster does not drain, and a VM live-migrates on drain
|
|
**only** with per-VM `evictionStrategy: LiveMigrate` — a blanket setting would wedge drains for
|
|
RWO-disk VMs.
|
|
- **Do not delete a VMI or `virt-launcher` pod** while the control plane is down: nothing will bring the
|
|
VM back until `virt-controller` is ready again.
|
|
- **Do not add the `app-namespace` label to `kubevirt` or `cdi`** to "harden" them.
|
|
- Escalate for vendoring the remote bases or for any operator upgrade. If the alert is a false positive,
|
|
fixing the rule is itself worth a change — a permanently-firing critical is a recorded, expensive
|
|
failure mode here.
|
|
|
|
---
|
|
Source of truth: `runbooks/kubevirt-control-plane.md`. Index: `/runbooks/`
|
|
Records: commits `b57b3e9` (this alert's wrong metric/job label) and `6a7e9d1` (false `CDIOperatorDown`);
|
|
`DECISIONS-LOG.md` (KubeVirt v1.9.0 + CDI v1.66.0 on nested KVM; the remote-base 27s fetch timeout).
|