feat(storage): CSI snapshots + smart-clone + migratable RWX for KubeVirt live-migration
Adds the VM/CDI storage tier approved by the PM (RWX for live-migration): - platform/external-snapshotter (NEW, controllers pass): the cluster-singleton snapshot.storage.k8s.io control plane (CRDs + snapshot-controller in kube-system), pinned v8.2.0 (Kubernetes deps aligned to 1.31 since v8.1.0; this cluster is k3s v1.31.5). These CRDs were absent — only snapshots.longhorn.io and kubevirt.io snapshot CRDs existed. - platform/longhorn/volumesnapshotclass.yaml (NEW, platform-config): the driver.longhorn.io VolumeSnapshotClass (type: snap, in-cluster). - platform/kubevirt/cdi-storageprofiles.yaml (NEW, platform-config): StorageProfile longhorn -> cloneStrategy: snapshot; StorageProfile longhorn-vm -> snapshot clone + claimPropertySets RWX/Block. Non-degraded default virt class clears CDIDefaultStorageClassDegraded. - platform/longhorn/storageclasses.yaml: NEW longhorn-vm StorageClass — migratable: "true" (Longhorn's KubeVirt-live-migration knob; RWX/Block-only on 1.7.3), is-default-virt-class (NOT the k8s default; longhorn stays RWO default). - platform/kubevirt/kubevirt-cr.yaml: comment only — LiveMigration is GA on v1.9 (no feature gate); no cluster-wide evictionStrategy (would wedge RWO-disk drains); live-migration stays per-VM opt-in on a longhorn-vm disk. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 4.8 committed
1 parent
b57b3e9d71
commit
3586769460
8 files changed
+215
-2
No files matched your search
@@ -29,6 +29,10 @@ resources:
|
||||
# cilium: L2 LB IP pool + announcement policy (cilium.io CRDs already registered by Cilium)
|
||||
- cilium/l2-ip-pool.yaml
|
||||
- cilium/l2-announcement-policy.yaml
|
||||
# longhorn/external-snapshotter: the driver.longhorn.io VolumeSnapshotClass (needs the
|
||||
# snapshot.storage.k8s.io CRDs from platform/external-snapshotter). Enables CSI snapshots +
|
||||
# CDI snapshot smart-clone.
|
||||
- ../platform/longhorn/volumesnapshotclass.yaml
|
||||
# monitoring: PrometheusRule alert rules (need monitoring.coreos.com CRDs from kube-prometheus-stack)
|
||||
- ../platform/monitoring/alerts.yaml
|
||||
# kubevirt/cdi (Phase 5): the KubeVirt + CDI CRs consume the kubevirt.io / cdi.kubevirt.io CRDs the
|
||||
@@ -37,6 +41,9 @@ resources:
|
||||
# ConfigMap kept beside them for locality.
|
||||
- ../platform/kubevirt/kubevirt-cr.yaml
|
||||
- ../platform/kubevirt/cdi-cr.yaml
|
||||
# CDI StorageProfiles: snapshot smart-clone for `longhorn`, and RWX/Block + snapshot clone for the
|
||||
# `longhorn-vm` virt-default class (consume cdi.kubevirt.io; clears CDIDefaultStorageClassDegraded).
|
||||
- ../platform/kubevirt/cdi-storageprofiles.yaml
|
||||
- ../platform/kubevirt/servicemonitor.yaml
|
||||
- ../platform/kubevirt/alerts.yaml
|
||||
- ../platform/kubevirt/dashboard.yaml
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
# external-snapshotter (CSI snapshot control plane)
|
||||
|
||||
Installs the cluster-singleton [kubernetes-csi/external-snapshotter](https://github.com/kubernetes-csi/external-snapshotter):
|
||||
|
||||
- the `snapshot.storage.k8s.io` CRDs — `VolumeSnapshot`, `VolumeSnapshotContent`,
|
||||
`VolumeSnapshotClass` (v1, GA);
|
||||
- the **snapshot-controller** Deployment (into `kube-system`, the upstream default) that
|
||||
reconciles those objects against any CSI driver's `CreateSnapshot`/`DeleteSnapshot`.
|
||||
|
||||
Neither k3s nor the Longhorn chart ships this. It is a prerequisite for:
|
||||
|
||||
1. **`VolumeSnapshotClass` (driver.longhorn.io)** — see
|
||||
`../longhorn/volumesnapshotclass.yaml` (applied by `platform-config`).
|
||||
2. **CDI snapshot smart-clone** — the CDI `StorageProfile`s set `cloneStrategy: snapshot`
|
||||
(see `../kubevirt/cdi-storageprofiles.yaml`); CDI clones VM disks by taking a CSI
|
||||
`VolumeSnapshot` instead of a slow host-assisted byte copy.
|
||||
3. Clearing the **`CDIDefaultStorageClassDegraded`** alert — a default virtualization
|
||||
StorageClass must support *both* smart-clone *and* ReadWriteMany.
|
||||
|
||||
## Wiring
|
||||
|
||||
- **Pass:** `platform` (controllers). Registers the CRDs.
|
||||
- The `VolumeSnapshotClass` / `StorageProfile` CRs that consume the CRDs live in the
|
||||
`platform-config` pass (`dependsOn: platform`), per the controllers/configs split.
|
||||
|
||||
## Version
|
||||
|
||||
Pinned to **v8.2.0** (Renovate-maintained). The v8.x line aligned to Kubernetes 1.31 at
|
||||
v8.1.0; this cluster is k3s v1.31.5. The snapshot-controller image is re-pinned to the same
|
||||
tag via the kustomization `images` transformer (the upstream setup manifest lags on the image
|
||||
tag).
|
||||
@@ -0,0 +1,44 @@
|
||||
---
|
||||
# external-snapshotter — the cluster-wide CSI snapshot control plane (Phase 2, storage).
|
||||
#
|
||||
# WHY: Longhorn's CSI driver (driver.longhorn.io) can create CSI VolumeSnapshots, but the
|
||||
# `snapshot.storage.k8s.io` API (VolumeSnapshot / VolumeSnapshotContent / VolumeSnapshotClass)
|
||||
# and the snapshot-controller that reconciles them are NOT part of k3s or of Longhorn — they are
|
||||
# a SEPARATE, cluster-singleton component (kubernetes-csi/external-snapshotter). Without it the
|
||||
# CRDs are absent (only `snapshots.longhorn.io` + the kubevirt.io snapshot CRDs exist), so:
|
||||
# * no VolumeSnapshotClass can be created (item 2), and
|
||||
# * CDI cannot do SNAPSHOT smart-clone (item 3) — it falls back to slow host-assisted `copy`,
|
||||
# which leaves the default virt StorageClass "degraded" (CDIDefaultStorageClassDegraded).
|
||||
# Installing it here (controllers pass) registers the CRDs; the VolumeSnapshotClass + the CDI
|
||||
# StorageProfiles that CONSUME them are applied by platform-config (dependsOn: platform).
|
||||
#
|
||||
# CONTROLLERS/CONFIGS SPLIT (F-MON1): this is the CONTROLLERS half. The snapshot.storage.k8s.io
|
||||
# CRDs are registered by the `client/config/crd` base below; the snapshot-controller Deployment
|
||||
# (into kube-system, upstream default) reconciles VolumeSnapshots. Custom resources that use these
|
||||
# CRDs (the Longhorn VolumeSnapshotClass, item 2) live in platform-config so the CRDs exist first.
|
||||
#
|
||||
# VERSION (Renovate-maintained): external-snapshotter v8.2.0.
|
||||
# * The v8.x line updated its Kubernetes deps to 1.31 at v8.1.0 (this cluster is k3s v1.31.5),
|
||||
# so v8.2.x is the conservative, version-matched choice — newer than the 1.31 alignment,
|
||||
# older than the v8.4/v8.5 churn.
|
||||
# * snapshot.storage.k8s.io CRDs are served at v1 (GA) — driver.longhorn.io speaks v1, so
|
||||
# Longhorn CSI snapshots + CDI snapshot-clone work against these CRDs.
|
||||
# * The upstream setup manifest pins the controller image to an older v8.0.x tag; the `images`
|
||||
# transformer below re-pins it to v8.2.0 so the running controller matches the CRD version.
|
||||
# * The `client/config/crd` base also ships the beta VolumeGroupSnapshot CRDs; that feature is
|
||||
# gated OFF in the controller (no --enable-volume-group-snapshots arg), so they are inert.
|
||||
#
|
||||
# Kustomize loads these as REMOTE bases (github.com/...?ref=<tag>) — the same remote-manifest
|
||||
# pattern platform/kubevirt uses for the KubeVirt/CDI operators; Flux's kustomize-controller
|
||||
# fetches them identically at reconcile.
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
# renovate: datasource=github-releases depName=kubernetes-csi/external-snapshotter
|
||||
- https://github.com/kubernetes-csi/external-snapshotter/client/config/crd?ref=v8.2.0
|
||||
# renovate: datasource=github-releases depName=kubernetes-csi/external-snapshotter
|
||||
- https://github.com/kubernetes-csi/external-snapshotter/deploy/kubernetes/snapshot-controller?ref=v8.2.0
|
||||
images:
|
||||
# Re-pin the controller image to the base ref so controller == CRD version (see header note).
|
||||
- name: registry.k8s.io/sig-storage/snapshot-controller
|
||||
newTag: v8.2.0
|
||||
@@ -0,0 +1,46 @@
|
||||
---
|
||||
# CDI StorageProfiles — the storage "recipes" CDI uses when it provisions/clones VM disk PVCs
|
||||
# (ADR-0005). CONSUME the cdi.kubevirt.io CRD registered by the CDI operator (platform pass), so —
|
||||
# like cdi-cr.yaml — these CRs are applied by the `platform-config` Flux Kustomization
|
||||
# (dependsOn: platform). They live here beside the KubeVirt/CDI component and are referenced in
|
||||
# place from platform-config/kustomization.yaml.
|
||||
#
|
||||
# CDI AUTO-CREATES a StorageProfile per StorageClass and fills its STATUS (inferred capabilities).
|
||||
# The operator owns `.status`; the USER owns `.spec`. Setting `.spec` here does NOT fight CDI — it
|
||||
# overrides only the fields we set (server-side apply), and CDI keeps reconciling status around them.
|
||||
#
|
||||
# ── longhorn: turn on SNAPSHOT smart-clone ─────────────────────────────────────────────────────
|
||||
# Default (empty spec) the longhorn profile reports `cloneStrategy: copy` — CDI clones a golden image
|
||||
# by spinning a host-assisted byte-for-byte copy pod (slow, doubles I/O). With external-snapshotter
|
||||
# installed and a driver.longhorn.io VolumeSnapshotClass present (../longhorn/volumesnapshotclass.yaml),
|
||||
# `cloneStrategy: snapshot` makes CDI clone by taking a CSI VolumeSnapshot and provisioning the new
|
||||
# PVC from it — near-instant, space-efficient. claimPropertySets left unset: CDI keeps inferring the
|
||||
# class's native RWO/Filesystem, correct for ordinary (non-migratable) longhorn PVCs.
|
||||
apiVersion: cdi.kubevirt.io/v1beta1
|
||||
kind: StorageProfile
|
||||
metadata:
|
||||
name: longhorn
|
||||
annotations:
|
||||
homelab.keithjackson.dev/adr: ADR-0005
|
||||
spec:
|
||||
cloneStrategy: snapshot
|
||||
---
|
||||
# ── longhorn-vm: SNAPSHOT smart-clone + RWX/Block for live-migratable VM disks ──────────────────
|
||||
# This is the default *virtualization* StorageClass (is-default-virt-class in ../longhorn/
|
||||
# storageclasses.yaml). Its profile must advertise the access/volume mode combo Longhorn 1.7.3
|
||||
# supports for migratable RWX — ReadWriteMany + Block — so every VM DataVolume CDI provisions on this
|
||||
# class is stamped RWX/Block and is therefore live-migratable. Together with cloneStrategy: snapshot
|
||||
# (smart clone) this profile is NON-DEGRADED (supports smart-clone AND RWX), which is what clears the
|
||||
# CDIDefaultStorageClassDegraded alert (the default virt class must support BOTH).
|
||||
apiVersion: cdi.kubevirt.io/v1beta1
|
||||
kind: StorageProfile
|
||||
metadata:
|
||||
name: longhorn-vm
|
||||
annotations:
|
||||
homelab.keithjackson.dev/adr: ADR-0005
|
||||
spec:
|
||||
cloneStrategy: snapshot
|
||||
claimPropertySets:
|
||||
- accessModes:
|
||||
- ReadWriteMany
|
||||
volumeMode: Block
|
||||
@@ -30,8 +30,18 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/adr: ADR-0005
|
||||
spec:
|
||||
# Drain VMs off a cordoned node when possible; software-emulated dev VMs just Evict+restart.
|
||||
# (Live migration across the 3 converged nodes is a nice-to-have per docs/06, not required.)
|
||||
# LIVE MIGRATION (docs/06): GA in KubeVirt since v0.43 — on v1.9 (this install) it is enabled by
|
||||
# default and needs NO feature gate. developerConfiguration.featureGates is intentionally left
|
||||
# EMPTY: the `LiveMigration` gate no longer exists as an alpha/beta toggle, and adding it would be
|
||||
# a no-op (worse, an unknown gate name can be rejected). No spec.configuration.migrations tuning is
|
||||
# added either — the defaults (parallelMigrationsPerCluster=5, parallelOutboundMigrationsPerNode=2,
|
||||
# 64Mi/s bandwidth cap) are correct for a 3-node dev cluster; revisit only under WS-VM load tests.
|
||||
#
|
||||
# A VM live-migrates on node drain ONLY if it opts in per-VM with `evictionStrategy: LiveMigrate`
|
||||
# AND its disks are on a RWX-migratable StorageClass — `longhorn-vm` (platform/longhorn/
|
||||
# storageclasses.yaml: migratable=true, RWX/Block). We do NOT set a cluster-wide
|
||||
# spec.configuration.evictionStrategy: a blanket LiveMigrate would WEDGE drains for VMs whose disks
|
||||
# are on the RWO `longhorn` class (unmigratable). VMs without opt-in still Evict+restart on drain.
|
||||
configuration:
|
||||
developerConfiguration:
|
||||
# Use real (nested) KVM. `false` makes virt-controller REQUEST the kvm device so /dev/kvm is
|
||||
|
||||
@@ -14,6 +14,7 @@ resources:
|
||||
- kyverno # admission enforcer FIRST — policies present before other components reconcile
|
||||
- cilium # CNI (adopts the running Ansible-installed release; parity verified)
|
||||
- longhorn # replicated block storage / CSI
|
||||
- external-snapshotter # CSI snapshot CRDs + snapshot-controller (VolumeSnapshot API; CDI smart-clone)
|
||||
- cert-manager # Let's Encrypt DNS-01 wildcard — needs SOPS `cloudflare-api-token`
|
||||
- externaldns # auto DNS records — needs SOPS `cloudflare-api-token`
|
||||
- gateway # Gateway API listeners (ingress; replaces Traefik)
|
||||
|
||||
@@ -29,6 +29,52 @@ parameters:
|
||||
dataLocality: "disabled" # keep replicas spread for durability, not co-located with the pod
|
||||
fsType: "ext4"
|
||||
---
|
||||
# KubeVirt VM-disk tier — RWX + migratable, for LIVE-MIGRATABLE VM disks (ADR-0005, docs/06).
|
||||
#
|
||||
# WHY a second Longhorn class: the default `longhorn` class is ReadWriteOnce (a Longhorn volume
|
||||
# attaches to ONE node). KubeVirt live migration needs the SOURCE and TARGET virt-launcher pods to
|
||||
# hold the SAME disk on TWO nodes for the migration window, which requires ReadWriteMany. Longhorn
|
||||
# provides this via the StorageClass parameter `migratable: "true"`.
|
||||
#
|
||||
# migratable RWX on Longhorn 1.7.3 is BLOCK-mode only (verified against Longhorn 1.7 docs + the live
|
||||
# cluster: longhornio/longhorn-manager:v1.7.3). Longhorn's ORDINARY RWX (migratable=false) is an
|
||||
# NFS/share-manager-backed *Filesystem* export; the MIGRATABLE RWX path is different — it is a raw
|
||||
# block device that Longhorn allows attached to two nodes ONLY during a live migration (concurrent
|
||||
# writers are not permitted, so a mounted filesystem could be corrupted). Therefore:
|
||||
# * StorageClass parameter migratable: "true"
|
||||
# * PVC/DataVolume accessModes ReadWriteMany
|
||||
# * PVC/DataVolume volumeMode Block (Filesystem is NOT supported with migratable)
|
||||
# The volumeMode/accessMode live on the PVC/DataVolume; CDI stamps them automatically for disks on
|
||||
# this class via the matching StorageProfile claimPropertySets (../kubevirt/cdi-storageprofiles.yaml).
|
||||
#
|
||||
# is-default-virt-class: CDI treats THIS as the default *virtualization* StorageClass (precedence over
|
||||
# the k8s default `longhorn` for VM disks when storageClassName is omitted). Because it supports smart
|
||||
# clone (snapshot) AND ReadWriteMany, its StorageProfile is non-degraded — which is what clears the
|
||||
# CDIDefaultStorageClassDegraded alert. It is NOT the cluster (k8s) default: `longhorn` keeps the
|
||||
# storageclass.kubernetes.io/is-default-class annotation, so ordinary PVCs still land on RWO longhorn.
|
||||
# Exactly one SC may carry is-default-virt-class (else CDIMultipleDefaultVirtStorageClasses fires).
|
||||
apiVersion: storage.k8s.io/v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: longhorn-vm
|
||||
annotations:
|
||||
storageclass.kubevirt.io/is-default-virt-class: "true"
|
||||
provisioner: driver.longhorn.io
|
||||
allowVolumeExpansion: true
|
||||
reclaimPolicy: Delete # dev: reclaim space when the PVC is deleted
|
||||
volumeBindingMode: Immediate
|
||||
parameters:
|
||||
# Same 3-replica durability as the default class (HARD node anti-affinity in helmrelease.yaml).
|
||||
# Quoted string literals — see the numberOfReplicas note on the `longhorn` class above.
|
||||
numberOfReplicas: "3"
|
||||
staleReplicaTimeout: "30"
|
||||
dataLocality: "disabled"
|
||||
# THE knob that makes this class usable for KubeVirt live migration: mark volumes migratable so a
|
||||
# RWX Longhorn volume can be attached to two nodes during a live migration. (Block-mode only.)
|
||||
migratable: "true"
|
||||
# No fsType: migratable RWX volumes are Block-mode (raw device); a filesystem is never laid down
|
||||
# by Longhorn here — the guest owns the block device.
|
||||
---
|
||||
# Bulk media tier — large, NON-replicated data (Plex library, download staging, arr handoff).
|
||||
# docs/04 / R-STATE-04: 3x replicating multi-TB media across 3 mini-PCs is impractical, so bulk
|
||||
# media lives OUTSIDE the cluster and is consumed as an external NFS/SMB mount via this thin
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
---
|
||||
# Longhorn VolumeSnapshotClass — the CSI snapshot recipe for driver.longhorn.io.
|
||||
#
|
||||
# CONSUMES the snapshot.storage.k8s.io CRD registered by platform/external-snapshotter (controllers
|
||||
# pass), so — like the KubeVirt/CDI CRs — this CR is applied by the `platform-config` Flux
|
||||
# Kustomization (dependsOn: platform), not in the controllers pass. It lives HERE beside the Longhorn
|
||||
# component for locality and is referenced in place from platform-config/kustomization.yaml.
|
||||
#
|
||||
# WHY: this is what turns "Longhorn has a CSI driver" into "the cluster can take CSI VolumeSnapshots
|
||||
# of Longhorn PVCs". CDI's snapshot smart-clone (cdi-storageprofiles.yaml -> cloneStrategy: snapshot)
|
||||
# resolves the driver's VolumeSnapshotClass by matching driver name; this is the only one for
|
||||
# driver.longhorn.io, so CDI selects it unambiguously (no is-default-class annotation needed).
|
||||
#
|
||||
# type: snap -> an in-cluster Longhorn snapshot on the volume's own replicas (fast, no external
|
||||
# backupstore required). The alternative `type: bak` uploads to a backup target, which
|
||||
# is not configured until Phase 4 (Velero/backupTarget). Clone/restore stay in-cluster.
|
||||
# deletionPolicy: Delete -> when the VolumeSnapshot is deleted, its Longhorn snapshot is cleaned up.
|
||||
# CDI's clone snapshots are transient, so leaking them would waste replica space.
|
||||
apiVersion: snapshot.storage.k8s.io/v1
|
||||
kind: VolumeSnapshotClass
|
||||
metadata:
|
||||
name: longhorn
|
||||
labels:
|
||||
app.kubernetes.io/part-of: longhorn
|
||||
driver: driver.longhorn.io
|
||||
deletionPolicy: Delete
|
||||
parameters:
|
||||
type: snap
|
||||
Reference in new issue
Block a user