jakman85andClaude Opus 5
44983095e1
Every platform-health alert rule carries `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations over 34 distinct names, across platform/monitoring/alerts.yaml, platform/monitoring/pvc-capacity-alerts.yaml, platform/cnpg/alerts.yaml and platform/kubevirt/alerts.yaml. There was no runbooks/ directory and no HTTPRoute serving that path: an in-cluster probe of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404. Every alert that reached a human handed them a dead link at the exact moment the documentation is read, and the annotation made it look solved — the same green-surface-over-absent-mechanism class as the import Job that reported success while storing placeholder secrets. No alerts.yaml is touched and no runbook_url value changes. The existing URL scheme is made to resolve instead. WHAT LANDS - runbooks/<34>.md — one per name, each grounded in this cluster's real topology and, where one exists, its real incident history: the 2026-09-07 k3s-prod-1 etcd I/O starvation (measured PSI 11.36 vs 0.75/1.29, the exact journal lines, the four mechanisms proven inert), the Longhorn reboot tax and the freeze/thaw of concurrent-replica-rebuild-per-node-limit, Cilium's once-resolved egress interface cache (Ifindex 3 against a live 1780) and the absence of OSS failover, the CNPG 38-hour ContinuousArchiving=False with its measured 6.8 MiB/h disk-full clock and the 2026-09-12 aggregation-scope fix, the 2026-09-06 Keycloak lockout in which all 12 clients held literal placeholder secrets behind an entirely green cluster, the Velero R2 x-amz-tagging NotImplemented incompatibility and its v1.14.3-rc.2 pin, and swap re-enabling itself across reboots via systemd-gpt-auto-generator. Each page states what the alert means HERE, the first diagnostic commands, likely causes ordered by probability with a remediation each, and when to escalate or stop. - Every page uses FULLY-QUALIFIED resource names and says why: three CRDs register the kind `Backup` here, the bare short name resolves to Longhorn's (zero objects), so `kubectl get backup -A` prints "No resources found" on a cluster holding 6 Velero and 27 CNPG backups. Acting on that false negative is how a CLI quirk becomes data loss. - Every page states that kubectl only works via `infra/devcontainer/dev.sh run bash -c '...'` with an explicit `--context k3s-prod`, because dev.sh defaults its current-context to k3s-dev. - Pages also record which alert groups are INERT today (measured 2026-09-13: no Longhorn, cert-manager, Flux-controller or Hubble scrape job exists, so 13 rules evaluate against nothing) so nobody reads their silence as health. SERVING — runbooks/serving/ plus clusters/base/runbooks.yaml A two-replica nginx-unprivileged serving a ConfigMap generated from runbooks/*.md, at the apex `/runbooks` prefix, in its own non-gating post-platform Flux tier (dependsOn platform-config, wait:false) — the platform/homepage pattern, file for file. - The service catalog cannot express this route: ingressRoutes[].host is pattern-bound to a single DNS label plus the apex, so the bare apex is unrepresentable, and there is no path field. - Rejected: redirecting to the Gitea web UI. A runbook is read when the platform is broken, and Gitea is an app-tier workload whose own dependencies several of these runbooks document; Gateway API's RequestRedirect also has no "append .md" modifier, so it would need 34 hand-maintained ReplaceFullPath rules. - configMapGenerator maps each file to an EXTENSIONLESS key (node-down=node-down.md), so /runbooks/node-down is a plain static lookup with no rewrite rule. Served as text/plain: the namespace has DNS-only egress, so a CDN markdown renderer could not load anyway. - The tier declares NO postBuild.substituteFrom, deliberately: envsubst would run over the prose and silently empty every ${...} shell example (it fails open — the 2026-09-01 emptied Cilium k8sServiceHost). The HTTPRoute omits `hostnames:` and inherits the https-apex listener's, so nothing needs substituting. AUTH — PUBLIC, deliberately auth-mode=none. Gating these pages behind oauth2-proxy/Keycloak would make runbooks/keycloak-down unreadable when Keycloak is down and runbooks/keycloak-client-secrets unreadable during the exact 2026-09-06 failure mode. An operator who cannot log in is this route's primary reader. Content is operational procedure only, no secrets. The rationale is recorded in an annotation on the HTTPRoute so the route is left OUT of the Gateway ext-authz attachment when forward-auth enforcement lands. PROVEN — in-cluster probe through the Gateway on prod /runbooks/node-down HTTP 200 8613 bytes text/plain; charset=utf-8 /runbooks/keycloak-down HTTP 200 9378 bytes text/plain; charset=utf-8 /runbooks/cnpg-archiving HTTP 200 11729 bytes text/plain; charset=utf-8 /runbooks/longhorn-faulted HTTP 200 9135 bytes text/plain; charset=utf-8 /runbooks/vpn-down HTTP 200 11092 bytes text/plain; charset=utf-8 /runbooks/backup-failure HTTP 200 9604 bytes text/plain; charset=utf-8 HTTPRoute Accepted by io.cilium/gateway-controller on sectionName https-apex; the `/runbooks` prefix wins over homepage's apex `/` by Gateway API longest-path precedence. GUARD — conformance/tests/l0-static/alert-runbooks-resolvable (R-OBS-04) 363 offline assertions. It checks BOTH halves, because the regression is silent in both directions: an alert with a new runbook_url and no runbook links to a 404, AND a runbook committed without its configMapGenerator entry is in git, reviewed, merged, and still 404s in the cluster. Also asserts a quality floor so a stub cannot make the link look fixed, and that route/listener/nginx prefix agree with the annotations. Proven to fail, then reverted: a bogus runbook_url (2 FAIL), a runbook removed from the generator (1 FAIL), and a 31-byte stub (7 FAIL). Co-Authored-By: Claude Opus 5 <[email protected]>
8.3 KiB
kubevirt-control-plane — The KubeVirt control plane is degraded
Alert: KubeVirtControlPlaneDown (critical, 10m)
File: platform/kubevirt/alerts.yaml, group platform-health.kubevirt
Expression:
(sum(kubevirt_virt_controller_ready_status) == 0)
or (sum(kubevirt_virt_api_ready_status) == 0)
or (absent(up{job="kubevirt-prometheus-metrics"} == 1))
What this means here
No virt-controller or no virt-api replica reports ready, so VM lifecycle operations — create, start,
stop, migrate — will fail (docs/06). Note carefully what this does not mean: already-running VMs
keep running. virt-launcher pods and their qemu processes are independent of the control plane. So
this is an "I cannot manage VMs" alert, not "the VMs are down".
The absent() clause also fires if the whole KubeVirt metrics target disappears. The scrape job label is
kubevirt-prometheus-metrics (the Service name), not kubevirt — that precision is a fix: this
very alert once used a non-existent metric with the wrong job label and fired falsely (commit b57b3e9).
There is a sibling precedent too: a broken HTTP CDI ServiceMonitor caused a false CDIOperatorDown
(commit 6a7e9d1). So the first question with this alert is always whether it is telling the truth.
Live state, measured 2026-09-13: kubevirt-prometheus-metrics is among Prometheus's 19 scrape jobs
and kubevirt_virt_api_ready_status returns 2 series, so the first two clauses are evaluable and
quiet. (kubevirt_vmi_phase_count was absent at the same moment — that metric only exists when VMIs do,
which affects /runbooks/kubevirt-launch-failed, not this rule.)
Platform context: KubeVirt v1.9.0 + CDI v1.66.0, on real nested KVM (/dev/kvm present, vmx/svm
verified; the Proxmox nodes use cpu_type=host). The kubevirt and cdi namespaces deliberately carry
no app-namespace label, so Kyverno does not generate the default-deny baseline into them — it would
sever the virt control plane. Do not "fix" that by adding the label.
Running the commands in this runbook
kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's k3s-oidc context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass --context k3s-prod explicitly. dev.sh merges both clusters into one kubeconfig and
defaults its current-context to k3s-dev, so a command without --context silently answers
about the wrong cluster. (CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... is equivalent.)
Do not run two dev.sh invocations at once — they share one container and interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup:
longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to
Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io,
settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.
Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for
agents here: every durable change goes through git (git push origin main is the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
First commands
Is the control plane actually unhealthy, or is this a metrics artefact?
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n kubevirt get pods -o wide; echo; \
kubectl --context k3s-prod get kubevirts.kubevirt.io -A \
-o custom-columns=NAME:.metadata.name,PHASE:.status.phase,READY:.status.conditions[?\(@.type==\"Available\"\)].status'
Are the VMs themselves still running? (This is the reassuring half.)
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get vmi -A -o wide; echo; \
kubectl --context k3s-prod get vm -A'
Is the scrape target up, and do the two metrics exist?
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
for q in "up{job=\"kubevirt-prometheus-metrics\"}" "sum(kubevirt_virt_controller_ready_status)" "sum(kubevirt_virt_api_ready_status)"; do \
printf "%-55s " "$q"; curl -s -G --data-urlencode "query=$q" localhost:19090/api/v1/query \
| python3 -c "import json,sys;print([x[\"value\"][1] for x in json.load(sys.stdin)[\"data\"][\"result\"]] or \"ABSENT\")"; done'
Why a component is unready:
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n kubevirt logs deploy/virt-controller --tail=80 2>&1 | tail -30; echo ====; \
kubectl --context k3s-prod -n kubevirt logs deploy/virt-api --tail=80 2>&1 | tail -30; echo ====; \
kubectl --context k3s-prod -n kubevirt get events --sort-by=.lastTimestamp | tail -20'
Is the Flux tier that owns KubeVirt healthy?
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \
-o custom-columns=NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message'
Likely causes, most likely first
1. The metrics target, not the control plane
Check the pods first. If virt-api and virt-controller are Running and Ready, this is a monitoring
fault — and there is direct precedent for false positives on this exact alert (b57b3e9) and its CDI
sibling (6a7e9d1). Remediation: fix the ServiceMonitor/metric in git. Cross-read
/runbooks/targets-down.
2. A node that hosted the control-plane pods went away
virt-controller and virt-api are ordinary Deployments. Go to /runbooks/node-down.
3. The KubeVirt Flux tier failed to reconcile
Specific known hazard: platform/kubevirt still uses remote kustomize bases, and the
kustomize-controller's git fetch has a 27s timeout that remote GitHub bases flake against — a
BuildFail there is probably this. external-snapshotter was vendored into the repo for exactly this
reason; vendoring KubeVirt the same way is the durable fix. See /runbooks/flux-reconcile.
4. Node pressure
Heavy sustained node load — TCG-emulated VMs are the recorded example — can degrade the Cilium datapath
into intermittent LB-VIP reachability problems, which looks like control-plane flapping. A restart
cleared it. Cross-read /runbooks/node-io and /runbooks/node-cpu.
5. A bad operator upgrade
/runbooks/update-rollback.
6. A NetworkPolicy severed the control plane
Only if someone added homelab.keithjackson.dev/app-namespace=true to kubevirt or cdi. Those
namespaces are deliberately unlabelled so Kyverno does not inject a default-deny baseline that would cut
virt-handler off. Reverting that label is the fix.
When to escalate or stop
- Prioritise honestly. VM lifecycle is broken; running VMs are not. The UniFi VM keeps serving. Treat this as urgent-but-not-catastrophic unless a VM also needs to start.
- Do not drain a node to move virt pods. This cluster does not drain, and a VM live-migrates on drain
only with per-VM
evictionStrategy: LiveMigrate— a blanket setting would wedge drains for RWO-disk VMs. - Do not delete a VMI or
virt-launcherpod while the control plane is down: nothing will bring the VM back untilvirt-controlleris ready again. - Do not add the
app-namespacelabel tokubevirtorcdito "harden" them. - Escalate for vendoring the remote bases or for any operator upgrade. If the alert is a false positive, fixing the rule is itself worth a change — a permanently-firing critical is a recorded, expensive failure mode here.
Source of truth: runbooks/kubevirt-control-plane.md. Index: /runbooks/
Records: commits b57b3e9 (this alert's wrong metric/job label) and 6a7e9d1 (false CDIOperatorDown);
DECISIONS-LOG.md (KubeVirt v1.9.0 + CDI v1.66.0 on nested KVM; the remote-base 27s fetch timeout).