feat(storage): CSI snapshots + smart-clone + migratable RWX for KubeVirt live-migration

Adds the VM/CDI storage tier approved by the PM (RWX for live-migration): - platform/external-snapshotter (NEW, controllers pass): the cluster-singleton snapshot.storage.k8s.io control plane (CRDs + snapshot-controller in kube-system), pinned v8.2.0 (Kubernetes deps aligned to 1.31 since v8.1.0; this cluster is k3s v1.31.5). These CRDs were absent — only snapshots.longhorn.io and kubevirt.io snapshot CRDs existed. - platform/longhorn/volumesnapshotclass.yaml (NEW, platform-config): the driver.longhorn.io VolumeSnapshotClass (type: snap, in-cluster). - platform/kubevirt/cdi-storageprofiles.yaml (NEW, platform-config): StorageProfile longhorn -> cloneStrategy: snapshot; StorageProfile longhorn-vm -> snapshot clone + claimPropertySets RWX/Block. Non-degraded default virt class clears CDIDefaultStorageClassDegraded. - platform/longhorn/storageclasses.yaml: NEW longhorn-vm StorageClass — migratable: "true" (Longhorn's KubeVirt-live-migration knob; RWX/Block-only on 1.7.3), is-default-virt-class (NOT the k8s default; longhorn stays RWO default). - platform/kubevirt/kubevirt-cr.yaml: comment only — LiveMigration is GA on v1.9 (no feature gate); no cluster-wide evictionStrategy (would wedge RWO-disk drains); live-migration stays per-VM opt-in on a longhorn-vm disk. Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 12:06:15 -05:00
1 parent b57b3e9d71
commit 3586769460
@@ -0,0 +1,31 @@
+# external-snapshotter (CSI snapshot control plane)
+
+Installs the cluster-singleton [kubernetes-csi/external-snapshotter](https://github.com/kubernetes-csi/external-snapshotter):
+
+- the `snapshot.storage.k8s.io` CRDs — `VolumeSnapshot`, `VolumeSnapshotContent`,
+  `VolumeSnapshotClass` (v1, GA);
+- the **snapshot-controller** Deployment (into `kube-system`, the upstream default) that
+  reconciles those objects against any CSI driver's `CreateSnapshot`/`DeleteSnapshot`.
+
+Neither k3s nor the Longhorn chart ships this. It is a prerequisite for:
+
+1. **`VolumeSnapshotClass` (driver.longhorn.io)** — see
+   `../longhorn/volumesnapshotclass.yaml` (applied by `platform-config`).
+2. **CDI snapshot smart-clone** — the CDI `StorageProfile`s set `cloneStrategy: snapshot`
+   (see `../kubevirt/cdi-storageprofiles.yaml`); CDI clones VM disks by taking a CSI
+   `VolumeSnapshot` instead of a slow host-assisted byte copy.
+3. Clearing the **`CDIDefaultStorageClassDegraded`** alert — a default virtualization
+   StorageClass must support *both* smart-clone *and* ReadWriteMany.
+
+## Wiring
+
+- **Pass:** `platform` (controllers). Registers the CRDs.
+- The `VolumeSnapshotClass` / `StorageProfile` CRs that consume the CRDs live in the
+  `platform-config` pass (`dependsOn: platform`), per the controllers/configs split.
+
+## Version
+
+Pinned to **v8.2.0** (Renovate-maintained). The v8.x line aligned to Kubernetes 1.31 at
+v8.1.0; this cluster is k3s v1.31.5. The snapshot-controller image is re-pinned to the same
+tag via the kustomization `images` transformer (the upstream setup manifest lags on the image
+tag).
@@ -0,0 +1,44 @@
+---
+# external-snapshotter — the cluster-wide CSI snapshot control plane (Phase 2, storage).
+#
+# WHY: Longhorn's CSI driver (driver.longhorn.io) can create CSI VolumeSnapshots, but the
+# `snapshot.storage.k8s.io` API (VolumeSnapshot / VolumeSnapshotContent / VolumeSnapshotClass)
+# and the snapshot-controller that reconciles them are NOT part of k3s or of Longhorn — they are
+# a SEPARATE, cluster-singleton component (kubernetes-csi/external-snapshotter). Without it the
+# CRDs are absent (only `snapshots.longhorn.io` + the kubevirt.io snapshot CRDs exist), so:
+#   * no VolumeSnapshotClass can be created (item 2), and
+#   * CDI cannot do SNAPSHOT smart-clone (item 3) — it falls back to slow host-assisted `copy`,
+#     which leaves the default virt StorageClass "degraded" (CDIDefaultStorageClassDegraded).
+# Installing it here (controllers pass) registers the CRDs; the VolumeSnapshotClass + the CDI
+# StorageProfiles that CONSUME them are applied by platform-config (dependsOn: platform).
+#
+# CONTROLLERS/CONFIGS SPLIT (F-MON1): this is the CONTROLLERS half. The snapshot.storage.k8s.io
+# CRDs are registered by the `client/config/crd` base below; the snapshot-controller Deployment
+# (into kube-system, upstream default) reconciles VolumeSnapshots. Custom resources that use these
+# CRDs (the Longhorn VolumeSnapshotClass, item 2) live in platform-config so the CRDs exist first.
+#
+# VERSION (Renovate-maintained): external-snapshotter v8.2.0.
+#   * The v8.x line updated its Kubernetes deps to 1.31 at v8.1.0 (this cluster is k3s v1.31.5),
+#     so v8.2.x is the conservative, version-matched choice — newer than the 1.31 alignment,
+#     older than the v8.4/v8.5 churn.
+#   * snapshot.storage.k8s.io CRDs are served at v1 (GA) — driver.longhorn.io speaks v1, so
+#     Longhorn CSI snapshots + CDI snapshot-clone work against these CRDs.
+#   * The upstream setup manifest pins the controller image to an older v8.0.x tag; the `images`
+#     transformer below re-pins it to v8.2.0 so the running controller matches the CRD version.
+#   * The `client/config/crd` base also ships the beta VolumeGroupSnapshot CRDs; that feature is
+#     gated OFF in the controller (no --enable-volume-group-snapshots arg), so they are inert.
+#
+# Kustomize loads these as REMOTE bases (github.com/...?ref=<tag>) — the same remote-manifest
+# pattern platform/kubevirt uses for the KubeVirt/CDI operators; Flux's kustomize-controller
+# fetches them identically at reconcile.
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  # renovate: datasource=github-releases depName=kubernetes-csi/external-snapshotter
+  - https://github.com/kubernetes-csi/external-snapshotter/client/config/crd?ref=v8.2.0
+  # renovate: datasource=github-releases depName=kubernetes-csi/external-snapshotter
+  - https://github.com/kubernetes-csi/external-snapshotter/deploy/kubernetes/snapshot-controller?ref=v8.2.0
+images:
+  # Re-pin the controller image to the base ref so controller == CRD version (see header note).
+  - name: registry.k8s.io/sig-storage/snapshot-controller
+    newTag: v8.2.0
@@ -0,0 +1,46 @@
+---
+# CDI StorageProfiles — the storage "recipes" CDI uses when it provisions/clones VM disk PVCs
+# (ADR-0005). CONSUME the cdi.kubevirt.io CRD registered by the CDI operator (platform pass), so —
+# like cdi-cr.yaml — these CRs are applied by the `platform-config` Flux Kustomization
+# (dependsOn: platform). They live here beside the KubeVirt/CDI component and are referenced in
+# place from platform-config/kustomization.yaml.
+#
+# CDI AUTO-CREATES a StorageProfile per StorageClass and fills its STATUS (inferred capabilities).
+# The operator owns `.status`; the USER owns `.spec`. Setting `.spec` here does NOT fight CDI — it
+# overrides only the fields we set (server-side apply), and CDI keeps reconciling status around them.
+#
+# ── longhorn: turn on SNAPSHOT smart-clone ─────────────────────────────────────────────────────
+# Default (empty spec) the longhorn profile reports `cloneStrategy: copy` — CDI clones a golden image
+# by spinning a host-assisted byte-for-byte copy pod (slow, doubles I/O). With external-snapshotter
+# installed and a driver.longhorn.io VolumeSnapshotClass present (../longhorn/volumesnapshotclass.yaml),
+# `cloneStrategy: snapshot` makes CDI clone by taking a CSI VolumeSnapshot and provisioning the new
+# PVC from it — near-instant, space-efficient. claimPropertySets left unset: CDI keeps inferring the
+# class's native RWO/Filesystem, correct for ordinary (non-migratable) longhorn PVCs.
+apiVersion: cdi.kubevirt.io/v1beta1
+kind: StorageProfile
+metadata:
+  name: longhorn
+  annotations:
+    homelab.keithjackson.dev/adr: ADR-0005
+spec:
+  cloneStrategy: snapshot
+---
+# ── longhorn-vm: SNAPSHOT smart-clone + RWX/Block for live-migratable VM disks ──────────────────
+# This is the default *virtualization* StorageClass (is-default-virt-class in ../longhorn/
+# storageclasses.yaml). Its profile must advertise the access/volume mode combo Longhorn 1.7.3
+# supports for migratable RWX — ReadWriteMany + Block — so every VM DataVolume CDI provisions on this
+# class is stamped RWX/Block and is therefore live-migratable. Together with cloneStrategy: snapshot
+# (smart clone) this profile is NON-DEGRADED (supports smart-clone AND RWX), which is what clears the
+# CDIDefaultStorageClassDegraded alert (the default virt class must support BOTH).
+apiVersion: cdi.kubevirt.io/v1beta1
+kind: StorageProfile
+metadata:
+  name: longhorn-vm
+  annotations:
+    homelab.keithjackson.dev/adr: ADR-0005
+spec:
+  cloneStrategy: snapshot
+  claimPropertySets:
+    - accessModes:
+        - ReadWriteMany
+      volumeMode: Block
@@ -30,8 +30,18 @@ metadata:
  annotations:
    homelab.keithjackson.dev/adr: ADR-0005
 spec:
-  # Drain VMs off a cordoned node when possible; software-emulated dev VMs just Evict+restart.
-  # (Live migration across the 3 converged nodes is a nice-to-have per docs/06, not required.)
+  # LIVE MIGRATION (docs/06): GA in KubeVirt since v0.43 — on v1.9 (this install) it is enabled by
+  # default and needs NO feature gate. developerConfiguration.featureGates is intentionally left
+  # EMPTY: the `LiveMigration` gate no longer exists as an alpha/beta toggle, and adding it would be
+  # a no-op (worse, an unknown gate name can be rejected). No spec.configuration.migrations tuning is
+  # added either — the defaults (parallelMigrationsPerCluster=5, parallelOutboundMigrationsPerNode=2,
+  # 64Mi/s bandwidth cap) are correct for a 3-node dev cluster; revisit only under WS-VM load tests.
+  #
+  # A VM live-migrates on node drain ONLY if it opts in per-VM with `evictionStrategy: LiveMigrate`
+  # AND its disks are on a RWX-migratable StorageClass — `longhorn-vm` (platform/longhorn/
+  # storageclasses.yaml: migratable=true, RWX/Block). We do NOT set a cluster-wide
+  # spec.configuration.evictionStrategy: a blanket LiveMigrate would WEDGE drains for VMs whose disks
+  # are on the RWO `longhorn` class (unmigratable). VMs without opt-in still Evict+restart on drain.
  configuration:
    developerConfiguration:
      # Use real (nested) KVM. `false` makes virt-controller REQUEST the kvm device so /dev/kvm is
@@ -14,6 +14,7 @@ resources:
  - kyverno            # admission enforcer FIRST — policies present before other components reconcile
  - cilium             # CNI (adopts the running Ansible-installed release; parity verified)
  - longhorn           # replicated block storage / CSI
+  - external-snapshotter  # CSI snapshot CRDs + snapshot-controller (VolumeSnapshot API; CDI smart-clone)
  - cert-manager       # Let's Encrypt DNS-01 wildcard — needs SOPS `cloudflare-api-token`
  - externaldns        # auto DNS records — needs SOPS `cloudflare-api-token`
  - gateway            # Gateway API listeners (ingress; replaces Traefik)
@@ -29,6 +29,52 @@ parameters:
  dataLocality: "disabled"       # keep replicas spread for durability, not co-located with the pod
  fsType: "ext4"
 ---
+# KubeVirt VM-disk tier — RWX + migratable, for LIVE-MIGRATABLE VM disks (ADR-0005, docs/06).
+#
+# WHY a second Longhorn class: the default `longhorn` class is ReadWriteOnce (a Longhorn volume
+# attaches to ONE node). KubeVirt live migration needs the SOURCE and TARGET virt-launcher pods to
+# hold the SAME disk on TWO nodes for the migration window, which requires ReadWriteMany. Longhorn
+# provides this via the StorageClass parameter `migratable: "true"`.
+#
+# migratable RWX on Longhorn 1.7.3 is BLOCK-mode only (verified against Longhorn 1.7 docs + the live
+# cluster: longhornio/longhorn-manager:v1.7.3). Longhorn's ORDINARY RWX (migratable=false) is an
+# NFS/share-manager-backed *Filesystem* export; the MIGRATABLE RWX path is different — it is a raw
+# block device that Longhorn allows attached to two nodes ONLY during a live migration (concurrent
+# writers are not permitted, so a mounted filesystem could be corrupted). Therefore:
+#   * StorageClass parameter        migratable: "true"
+#   * PVC/DataVolume accessModes     ReadWriteMany
+#   * PVC/DataVolume volumeMode      Block            (Filesystem is NOT supported with migratable)
+# The volumeMode/accessMode live on the PVC/DataVolume; CDI stamps them automatically for disks on
+# this class via the matching StorageProfile claimPropertySets (../kubevirt/cdi-storageprofiles.yaml).
+#
+# is-default-virt-class: CDI treats THIS as the default *virtualization* StorageClass (precedence over
+# the k8s default `longhorn` for VM disks when storageClassName is omitted). Because it supports smart
+# clone (snapshot) AND ReadWriteMany, its StorageProfile is non-degraded — which is what clears the
+# CDIDefaultStorageClassDegraded alert. It is NOT the cluster (k8s) default: `longhorn` keeps the
+# storageclass.kubernetes.io/is-default-class annotation, so ordinary PVCs still land on RWO longhorn.
+# Exactly one SC may carry is-default-virt-class (else CDIMultipleDefaultVirtStorageClasses fires).
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: longhorn-vm
+  annotations:
+    storageclass.kubevirt.io/is-default-virt-class: "true"
+provisioner: driver.longhorn.io
+allowVolumeExpansion: true
+reclaimPolicy: Delete            # dev: reclaim space when the PVC is deleted
+volumeBindingMode: Immediate
+parameters:
+  # Same 3-replica durability as the default class (HARD node anti-affinity in helmrelease.yaml).
+  # Quoted string literals — see the numberOfReplicas note on the `longhorn` class above.
+  numberOfReplicas: "3"
+  staleReplicaTimeout: "30"
+  dataLocality: "disabled"
+  # THE knob that makes this class usable for KubeVirt live migration: mark volumes migratable so a
+  # RWX Longhorn volume can be attached to two nodes during a live migration. (Block-mode only.)
+  migratable: "true"
+  # No fsType: migratable RWX volumes are Block-mode (raw device); a filesystem is never laid down
+  # by Longhorn here — the guest owns the block device.
+---
 # Bulk media tier — large, NON-replicated data (Plex library, download staging, arr handoff).
 # docs/04 / R-STATE-04: 3x replicating multi-TB media across 3 mini-PCs is impractical, so bulk
 # media lives OUTSIDE the cluster and is consumed as an external NFS/SMB mount via this thin
@@ -0,0 +1,28 @@
+---
+# Longhorn VolumeSnapshotClass — the CSI snapshot recipe for driver.longhorn.io.
+#
+# CONSUMES the snapshot.storage.k8s.io CRD registered by platform/external-snapshotter (controllers
+# pass), so — like the KubeVirt/CDI CRs — this CR is applied by the `platform-config` Flux
+# Kustomization (dependsOn: platform), not in the controllers pass. It lives HERE beside the Longhorn
+# component for locality and is referenced in place from platform-config/kustomization.yaml.
+#
+# WHY: this is what turns "Longhorn has a CSI driver" into "the cluster can take CSI VolumeSnapshots
+# of Longhorn PVCs". CDI's snapshot smart-clone (cdi-storageprofiles.yaml -> cloneStrategy: snapshot)
+# resolves the driver's VolumeSnapshotClass by matching driver name; this is the only one for
+# driver.longhorn.io, so CDI selects it unambiguously (no is-default-class annotation needed).
+#
+# type: snap  -> an in-cluster Longhorn snapshot on the volume's own replicas (fast, no external
+#                backupstore required). The alternative `type: bak` uploads to a backup target, which
+#                is not configured until Phase 4 (Velero/backupTarget). Clone/restore stay in-cluster.
+# deletionPolicy: Delete -> when the VolumeSnapshot is deleted, its Longhorn snapshot is cleaned up.
+#                CDI's clone snapshots are transient, so leaking them would waste replica space.
+apiVersion: snapshot.storage.k8s.io/v1
+kind: VolumeSnapshotClass
+metadata:
+  name: longhorn
+  labels:
+    app.kubernetes.io/part-of: longhorn
+driver: driver.longhorn.io
+deletionPolicy: Delete
+parameters:
+  type: snap