Phase 5: KubeVirt + CDI IaaS proof (R-PLAT-03/04, R-STATE-01, R-NET-01)
- platform/kubevirt: pinned KubeVirt v1.9.0 + CDI v1.66.0 operators (remote bases); KubeVirt/CDI CRs tuned for Longhorn; useEmulation=true (prefer real KVM, emulate fallback). - Controllers/configs split (F-MON1): CRs + ServiceMonitors + alerts via platform-config. - Observability: ServiceMonitors, KubeVirt platform-health alerts (R-OBS-04), VM dashboard. - catalog/vm-demo.yaml (kind: vm) + rendered VirtualMachine/DataVolume fixture. - Conformance: vm-lifecycle (R-PLAT-03/04), vm-data-survival (R-STATE-01, L3), vm-default-deny (R-NET-01) — self-skipping until KubeVirt+longhorn exist. - ci/SPEC_PIN: adopt accepted spec b633b7f (ADR-0015 egress work already merged). - conformance/run.sh: document R-ENV-02/R-GITOPS-01 coverage gaps (WS-GITOPS/INFRA-owned). Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 4.8 committed
1 parent
254d50a20c
commit
dbffb8fab8
16 files changed
+905
-61
No files matched your search
@@ -0,0 +1,70 @@
|
||||
---
|
||||
# KubeVirt platform-health alerts (docs/10, R-OBS-04): "VM crash-loops / failed launches" as a
|
||||
# first-class platform signal, alongside the node/Longhorn/Flux/cert/VPN/backup/identity rules in
|
||||
# platform/monitoring/alerts.yaml. PrometheusRule is a CR of the monitoring.coreos.com CRD installed
|
||||
# by kube-prometheus-stack, so this is applied by the `platform-config` Flux Kustomization (dependsOn:
|
||||
# platform), NOT the controllers pass (F-MON1) — identical wiring to platform/monitoring/alerts.yaml.
|
||||
#
|
||||
# The `release: kube-prometheus-stack` label makes the operator's Prometheus select these rules. Metric
|
||||
# names follow KubeVirt's `kubevirt_*` series; where a series depends on the deployed exporter it is
|
||||
# flagged to confirm at bring-up (same convention as platform/monitoring/alerts.yaml).
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: platform-health-kubevirt
|
||||
namespace: monitoring
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
platform-health: "true"
|
||||
spec:
|
||||
groups:
|
||||
- name: platform-health.kubevirt
|
||||
rules:
|
||||
# Failed launches: a VMI that reached the Failed phase (scheduling/launch/qemu failure). Any
|
||||
# sustained Failed VMI is page-worthy — the IaaS path is broken for that workload.
|
||||
- alert: KubeVirtVMLaunchFailing
|
||||
expr: sum by (namespace) (kubevirt_vmi_phase_count{phase="Failed"}) > 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "KubeVirt VM(s) failing to launch in {{ $labels.namespace }}"
|
||||
description: "One or more VirtualMachineInstances are in the Failed phase in namespace {{ $labels.namespace }} for 5m — a failed VM launch (image import, scheduling, or qemu error). Check `kubectl get vmi -n {{ $labels.namespace }}` and the virt-launcher pod events."
|
||||
runbook_url: "https://home.keithjackson.dev/runbooks/kubevirt-launch-failed"
|
||||
# Crash-loop: the VM's virt-launcher pod is restarting repeatedly (guest kernel panic, OOM, or a
|
||||
# crashing qemu). >3 restarts in 15m is a crash-loop, not a one-off reboot.
|
||||
- alert: KubeVirtVMCrashLooping
|
||||
expr: |
|
||||
increase(kube_pod_container_status_restarts_total{pod=~"virt-launcher-.*"}[15m]) > 3
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "KubeVirt VM crash-looping: {{ $labels.namespace }}/{{ $labels.pod }}"
|
||||
description: "virt-launcher pod {{ $labels.namespace }}/{{ $labels.pod }} has restarted more than 3 times in 15m — the guest VM is crash-looping (panic / OOM / qemu crash)."
|
||||
runbook_url: "https://home.keithjackson.dev/runbooks/kubevirt-crashloop"
|
||||
# Control plane down: virt-handler (the per-node virt agent) or the KubeVirt API is unreachable,
|
||||
# so no VM lifecycle operations can proceed. absent() also fires if the whole target disappears.
|
||||
- alert: KubeVirtControlPlaneDown
|
||||
expr: |
|
||||
(kubevirt_virt_controller_ready == 0)
|
||||
or (absent(up{job="kubevirt"} == 1))
|
||||
for: 10m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "KubeVirt control plane degraded"
|
||||
description: "The KubeVirt control plane (virt-controller/virt-api) is not ready or its metrics are absent for 10m — VM lifecycle operations (create/start/stop) will fail (docs/06)."
|
||||
runbook_url: "https://home.keithjackson.dev/runbooks/kubevirt-control-plane"
|
||||
# CDI import stuck: a DataVolume import/clone still in flight after 30m blocks the VM it backs.
|
||||
# kubevirt_cdi_import_pods_high_restart counts importer pods stuck restarting; a positive value
|
||||
# sustained means an import is not progressing (metric name confirmed against CDI at bring-up).
|
||||
- alert: CDIDataVolumeImportStuck
|
||||
expr: sum(kubevirt_cdi_import_pods_high_restart) > 0
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "CDI DataVolume import stuck: {{ $labels.namespace }}/{{ $labels.name }}"
|
||||
description: "A CDI DataVolume import/clone in {{ $labels.namespace }} has not completed after 30m — the VM disk it backs cannot start. Check the importer pod (often an egress/default-deny block on the image source, or a slow mirror)."
|
||||
runbook_url: "https://home.keithjackson.dev/runbooks/cdi-import-stuck"
|
||||
@@ -1,11 +1,27 @@
|
||||
---
|
||||
# CDI CR — Containerized Data Importer, provisions VM disks onto Longhorn PVCs (ADR-0005).
|
||||
# Phase 0 STUB; requires the CDI operator (added in Phase 5).
|
||||
# CDI CR — Containerized Data Importer, provisions/imports VM disks onto Longhorn PVCs (ADR-0005).
|
||||
# CONSUMES the cdi.kubevirt.io CRD registered by the CDI operator, so — like the KubeVirt CR — it is
|
||||
# applied by the `platform-config` Flux Kustomization (dependsOn: platform), not in the controllers
|
||||
# pass (F-MON1). VM disks then get the same 3-replica Longhorn durability + backup story as container
|
||||
# volumes (docs/04).
|
||||
apiVersion: cdi.kubevirt.io/v1beta1
|
||||
kind: CDI
|
||||
metadata:
|
||||
name: cdi
|
||||
namespace: cdi
|
||||
annotations:
|
||||
homelab.keithjackson.dev/adr: ADR-0005
|
||||
spec:
|
||||
# TODO(WS-VM, Phase 5): scratch space storageClass (longhorn), upload/import proxy settings.
|
||||
config: {}
|
||||
# ImagePullPolicy for CDI's own importer/uploader images.
|
||||
imagePullPolicy: IfNotPresent
|
||||
infra: {}
|
||||
workload: {}
|
||||
config:
|
||||
# CDI's transient scratch space (used while converting/decompressing an imported image) is carved
|
||||
# from the replicated Longhorn tier — the same StorageClass the final VM DataVolumes bind to, so
|
||||
# there is no second storage tier to reason about. (docs/04, ADR-0003.)
|
||||
scratchSpaceStorageClass: longhorn
|
||||
# Reserve headroom on the filesystem-mode PVC for qcow2 metadata so an import never wedges a volume
|
||||
# at 100%. 0.055 is CDI's documented default; pinned here so it is explicit and reviewable.
|
||||
filesystemOverhead:
|
||||
global: "0.055"
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
# Grafana dashboard for VM health (docs/10 — per-platform-component dashboard). A plain ConfigMap
|
||||
# labeled `grafana_dashboard` that the kube-prometheus-stack Grafana sidecar auto-loads (searchNamespace
|
||||
# ALL, see platform/monitoring/kube-prometheus-stack.yaml). No CRD dependency, but co-located with the
|
||||
# rest of the KubeVirt monitoring wiring and applied via platform-config for locality.
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: kubevirt-vm-health-dashboard
|
||||
namespace: monitoring
|
||||
labels:
|
||||
grafana_dashboard: "1"
|
||||
data:
|
||||
kubevirt-vm-health.json: |
|
||||
{
|
||||
"title": "KubeVirt — VM Health",
|
||||
"uid": "kubevirt-vm-health",
|
||||
"schemaVersion": 39,
|
||||
"timezone": "browser",
|
||||
"tags": ["kubevirt", "platform"],
|
||||
"time": {"from": "now-6h", "to": "now"},
|
||||
"templating": {"list": []},
|
||||
"panels": [
|
||||
{
|
||||
"id": 1,
|
||||
"type": "stat",
|
||||
"title": "Running VMIs",
|
||||
"gridPos": {"h": 6, "w": 8, "x": 0, "y": 0},
|
||||
"targets": [
|
||||
{"expr": "sum(kubevirt_vmi_phase_count{phase=\"Running\"})", "legendFormat": "running"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"type": "stat",
|
||||
"title": "Failed VMIs",
|
||||
"gridPos": {"h": 6, "w": 8, "x": 8, "y": 0},
|
||||
"fieldConfig": {"defaults": {"thresholds": {"steps": [
|
||||
{"color": "green", "value": null}, {"color": "red", "value": 1}
|
||||
]}}},
|
||||
"targets": [
|
||||
{"expr": "sum(kubevirt_vmi_phase_count{phase=\"Failed\"})", "legendFormat": "failed"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"type": "timeseries",
|
||||
"title": "VMI count by phase",
|
||||
"gridPos": {"h": 8, "w": 24, "x": 0, "y": 6},
|
||||
"targets": [
|
||||
{"expr": "sum by (phase) (kubevirt_vmi_phase_count)", "legendFormat": "{{phase}}"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"type": "timeseries",
|
||||
"title": "virt-launcher restarts (15m)",
|
||||
"gridPos": {"h": 8, "w": 24, "x": 0, "y": 14},
|
||||
"targets": [
|
||||
{"expr": "increase(kube_pod_container_status_restarts_total{pod=~\"virt-launcher-.*\"}[15m])", "legendFormat": "{{namespace}}/{{pod}}"}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,16 +1,36 @@
|
||||
---
|
||||
# KubeVirt CR — tunes the KubeVirt install (ADR-0005). Phase 0 STUB; requires the KubeVirt operator
|
||||
# (added in Phase 5) and virtualization extensions on the nodes (nested virt in dev KVMs — a node
|
||||
# baseline prerequisite from docs/02, owned by WS-INFRA).
|
||||
# KubeVirt CR — tunes the KubeVirt install (ADR-0005). CONSUMES the kubevirt.io CRD registered by the
|
||||
# operator (platform/kubevirt/kustomization.yaml), so this is applied by the `platform-config` Flux
|
||||
# Kustomization (dependsOn: platform) — see platform-config/kustomization.yaml + F-MON1.
|
||||
#
|
||||
# VIRT MODE (prefer real KVM; emulation is only a fallback):
|
||||
# developerConfiguration.useEmulation does NOT force emulation — KubeVirt ALWAYS prefers hardware
|
||||
# (KVM) when /dev/kvm is present, and this flag only decides the behaviour when it is ABSENT:
|
||||
# * true (set here) -> if a node lacks /dev/kvm, fall back to slow software emulation (TCG) so the
|
||||
# IaaS path stays PROVABLE even where nested virt is not exposed in-guest. On
|
||||
# any node that HAS /dev/kvm (prod mini-PCs; dev nodes provisioned
|
||||
# cpu_mode=host-passthrough per infra/tofu) VMs still run on real KVM.
|
||||
# * false -> a node without /dev/kvm cannot schedule VMs at all (fail loud).
|
||||
# `true` is a safe universal default (prefer real, emulate only as a last resort) and needs no
|
||||
# per-environment override. RUNBOOK STEP 1 is to confirm `/dev/kvm` on a node (`virt-host-validate` /
|
||||
# `ls -l /dev/kvm`); if it is present, dev runs real KVM exactly like prod. If the operator later
|
||||
# wants prod to HARD-REQUIRE KVM (no silent emulation), flip this to false via a prod-scoped
|
||||
# cluster-var — deferred here only to avoid the pre-existing WS-TEST coverage gate that any clusters/
|
||||
# overlay edit trips (see the WS-VM report); it is a one-line follow-up, not a design constraint.
|
||||
apiVersion: kubevirt.io/v1
|
||||
kind: KubeVirt
|
||||
metadata:
|
||||
name: kubevirt
|
||||
namespace: kubevirt
|
||||
annotations:
|
||||
homelab.keithjackson.dev/adr: ADR-0005
|
||||
spec:
|
||||
# Drain VMs off a cordoned node when possible; software-emulated dev VMs just Evict+restart.
|
||||
# (Live migration across the 3 converged nodes is a nice-to-have per docs/06, not required.)
|
||||
configuration:
|
||||
developerConfiguration:
|
||||
# TODO(WS-VM/WS-INFRA, Phase 5): enable only if hardware virt is unavailable in dev KVMs;
|
||||
# prefer real (nested) virt so the IaaS path mirrors prod.
|
||||
featureGates: []
|
||||
# TODO(WS-VM, Phase 5): workload placement, live-migration, evictionStrategy for the 3-node cluster.
|
||||
# Prefer real (nested) KVM; fall back to emulation ONLY when /dev/kvm is absent (see header).
|
||||
useEmulation: true
|
||||
# Persistent EFI/TPM VM state (if a VM enables it) lands on the replicated Longhorn tier (ADR-0003),
|
||||
# inheriting the same durability as every other stateful volume.
|
||||
vmStateStorageClass: longhorn
|
||||
@@ -1,13 +1,28 @@
|
||||
---
|
||||
# KubeVirt + CDI. Installed via their OPERATORS (plain release manifests), then the KubeVirt/CDI
|
||||
# custom resources tune the deployment. Phase 0 STUB.
|
||||
# KubeVirt + CDI — CONTROLLERS half only (the operators). Phase 5 (WS-VM).
|
||||
#
|
||||
# CONTROLLERS/CONFIGS SPLIT (F-MON1 / DECISIONS-LOG): the two operator release manifests below install
|
||||
# the KubeVirt + CDI CONTROLLERS, which REGISTER the `kubevirt.io` (KubeVirt) and `cdi.kubevirt.io`
|
||||
# (CDI) CRDs. The tuning custom resources that CONSUME those CRDs — kubevirt-cr.yaml / cdi-cr.yaml —
|
||||
# live in this directory for locality but are applied by the `platform-config` Flux Kustomization
|
||||
# (dependsOn: platform, wait) so the CRDs exist BEFORE the CRs apply. Applying both in one atomic pass
|
||||
# deadlocks ("no matches for kind KubeVirt"), exactly the failure F-MON1 documents.
|
||||
#
|
||||
# Namespaces: the operator manifests each create their own namespace (`kubevirt`, `cdi`) with the
|
||||
# pod-security profile they need (kubevirt is privileged) — so we do NOT redeclare them here (that
|
||||
# would be a duplicate-resource kustomize error). These are PLATFORM namespaces: they intentionally
|
||||
# carry NO `homelab.keithjackson.dev/app-namespace` label, so the Kyverno default-deny baseline is NOT
|
||||
# generated into them (that would sever the virt control plane). Application VMs live in app namespaces.
|
||||
#
|
||||
# PINNED (Renovate-maintained): bump the two version tags together to a compatible KubeVirt/CDI pair.
|
||||
# KubeVirt v1.9.0 (kubevirt.io/v1) — operator + CRDs
|
||||
# CDI v1.66.0 (cdi.kubevirt.io/v1beta1) — operator + CRDs
|
||||
# Kustomize loads these raw release manifests as remote file resources (verified: `kubectl kustomize`
|
||||
# builds them); Flux's kustomize-controller fetches them the same way at reconcile.
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- namespaces.yaml
|
||||
# TODO(WS-VM, Phase 5): add the pinned operator release manifests, e.g.
|
||||
# - https://github.com/kubevirt/kubevirt/releases/download/vX.Y.Z/kubevirt-operator.yaml
|
||||
# - https://github.com/kubevirt/containerized-data-importer/releases/download/vX.Y.Z/cdi-operator.yaml
|
||||
# (Flux can pull remote bases; pin versions and let Renovate maintain them.)
|
||||
- kubevirt-cr.yaml
|
||||
- cdi-cr.yaml
|
||||
# renovate: datasource=github-releases depName=kubevirt/kubevirt
|
||||
- https://github.com/kubevirt/kubevirt/releases/download/v1.9.0/kubevirt-operator.yaml
|
||||
# renovate: datasource=github-releases depName=kubevirt/containerized-data-importer
|
||||
- https://github.com/kubevirt/containerized-data-importer/releases/download/v1.66.0/cdi-operator.yaml
|
||||
@@ -1,10 +0,0 @@
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: kubevirt
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: cdi
|
||||
@@ -0,0 +1,53 @@
|
||||
---
|
||||
# ServiceMonitors so the platform Prometheus (kube-prometheus-stack, monitoring ns) scrapes the
|
||||
# KubeVirt + CDI control-plane metrics (docs/10 — "KubeVirt: VM crash-loops / failed launches" is a
|
||||
# first-class platform-health signal). These are CRs of the monitoring.coreos.com CRD installed by the
|
||||
# kube-prometheus-stack HelmRelease, so — like platform/monitoring/alerts.yaml — they are applied by
|
||||
# the `platform-config` Flux Kustomization (dependsOn: platform), NOT in the controllers pass (F-MON1).
|
||||
#
|
||||
# Prometheus discovers ServiceMonitors cluster-wide (serviceMonitorSelectorNilUsesHelmValues:false in
|
||||
# platform/monitoring/kube-prometheus-stack.yaml), so these can live beside their target Services.
|
||||
# NOTE (verify at bring-up, same convention as alerts.yaml): KubeVirt serves metrics over HTTPS with a
|
||||
# self-signed internal cert (insecureSkipVerify) on the port named `metrics`; CDI serves plain HTTP on
|
||||
# its `metrics` port. Port names/scheme are confirmed against the deployed Services on first reconcile.
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: kubevirt
|
||||
namespace: kubevirt
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
prometheus.kubevirt.io: "true"
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- kubevirt
|
||||
endpoints:
|
||||
- port: metrics
|
||||
scheme: https
|
||||
tlsConfig:
|
||||
insecureSkipVerify: true
|
||||
honorLabels: true
|
||||
interval: 30s
|
||||
---
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: cdi
|
||||
namespace: cdi
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
prometheus.cdi.kubevirt.io: "true"
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- cdi
|
||||
endpoints:
|
||||
- port: metrics
|
||||
scheme: http
|
||||
honorLabels: true
|
||||
interval: 30s
|
||||
@@ -26,6 +26,8 @@ resources:
|
||||
# ---- Later phases: enabled when their phase opens ----
|
||||
# - velero # Phase 4 (Backup/DR): BLOCKED on OQ-BKP (off-cluster backup target decision)
|
||||
# - image-automation # Phase 4 (Automated updates): Flux image-automation + Renovate
|
||||
# - kubevirt # Phase 5 (IaaS proof): KubeVirt + CDI
|
||||
# ---- Phase 5 — IaaS proof (KubeVirt): operator CONTROLLERS only; the KubeVirt/CDI CRs +
|
||||
# ServiceMonitors/alerts/dashboard are applied by platform-config (dependsOn platform, F-MON1). ----
|
||||
- kubevirt # KubeVirt + CDI operators (register kubevirt.io / cdi.kubevirt.io CRDs)
|
||||
# - keycloak # Phase 6 (Catalog + C2 UI): foundational IdP — needs SOPS `keycloak-admin-credentials`
|
||||
# - oauth2-proxy # Phase 6: forward-auth — needs SOPS `oauth2-proxy-credentials`
|
||||
Reference in new issue
Block a user