diff --git a/platform-config/kustomization.yaml b/platform-config/kustomization.yaml index ca9bfc1..c4408c2 100644 --- a/platform-config/kustomization.yaml +++ b/platform-config/kustomization.yaml @@ -29,6 +29,10 @@ resources: # cilium: L2 LB IP pool + announcement policy (cilium.io CRDs already registered by Cilium) - cilium/l2-ip-pool.yaml - cilium/l2-announcement-policy.yaml + # longhorn/external-snapshotter: the driver.longhorn.io VolumeSnapshotClass (needs the + # snapshot.storage.k8s.io CRDs from platform/external-snapshotter). Enables CSI snapshots + + # CDI snapshot smart-clone. + - ../platform/longhorn/volumesnapshotclass.yaml # monitoring: PrometheusRule alert rules (need monitoring.coreos.com CRDs from kube-prometheus-stack) - ../platform/monitoring/alerts.yaml # kubevirt/cdi (Phase 5): the KubeVirt + CDI CRs consume the kubevirt.io / cdi.kubevirt.io CRDs the @@ -37,6 +41,9 @@ resources: # ConfigMap kept beside them for locality. - ../platform/kubevirt/kubevirt-cr.yaml - ../platform/kubevirt/cdi-cr.yaml + # CDI StorageProfiles: snapshot smart-clone for `longhorn`, and RWX/Block + snapshot clone for the + # `longhorn-vm` virt-default class (consume cdi.kubevirt.io; clears CDIDefaultStorageClassDegraded). + - ../platform/kubevirt/cdi-storageprofiles.yaml - ../platform/kubevirt/servicemonitor.yaml - ../platform/kubevirt/alerts.yaml - ../platform/kubevirt/dashboard.yaml diff --git a/platform/external-snapshotter/README.md b/platform/external-snapshotter/README.md new file mode 100644 index 0000000..b972c7b --- /dev/null +++ b/platform/external-snapshotter/README.md @@ -0,0 +1,31 @@ +# external-snapshotter (CSI snapshot control plane) + +Installs the cluster-singleton [kubernetes-csi/external-snapshotter](https://github.com/kubernetes-csi/external-snapshotter): + +- the `snapshot.storage.k8s.io` CRDs — `VolumeSnapshot`, `VolumeSnapshotContent`, + `VolumeSnapshotClass` (v1, GA); +- the **snapshot-controller** Deployment (into `kube-system`, the upstream default) that + reconciles those objects against any CSI driver's `CreateSnapshot`/`DeleteSnapshot`. + +Neither k3s nor the Longhorn chart ships this. It is a prerequisite for: + +1. **`VolumeSnapshotClass` (driver.longhorn.io)** — see + `../longhorn/volumesnapshotclass.yaml` (applied by `platform-config`). +2. **CDI snapshot smart-clone** — the CDI `StorageProfile`s set `cloneStrategy: snapshot` + (see `../kubevirt/cdi-storageprofiles.yaml`); CDI clones VM disks by taking a CSI + `VolumeSnapshot` instead of a slow host-assisted byte copy. +3. Clearing the **`CDIDefaultStorageClassDegraded`** alert — a default virtualization + StorageClass must support *both* smart-clone *and* ReadWriteMany. + +## Wiring + +- **Pass:** `platform` (controllers). Registers the CRDs. +- The `VolumeSnapshotClass` / `StorageProfile` CRs that consume the CRDs live in the + `platform-config` pass (`dependsOn: platform`), per the controllers/configs split. + +## Version + +Pinned to **v8.2.0** (Renovate-maintained). The v8.x line aligned to Kubernetes 1.31 at +v8.1.0; this cluster is k3s v1.31.5. The snapshot-controller image is re-pinned to the same +tag via the kustomization `images` transformer (the upstream setup manifest lags on the image +tag). diff --git a/platform/external-snapshotter/kustomization.yaml b/platform/external-snapshotter/kustomization.yaml new file mode 100644 index 0000000..12050b0 --- /dev/null +++ b/platform/external-snapshotter/kustomization.yaml @@ -0,0 +1,44 @@ +--- +# external-snapshotter — the cluster-wide CSI snapshot control plane (Phase 2, storage). +# +# WHY: Longhorn's CSI driver (driver.longhorn.io) can create CSI VolumeSnapshots, but the +# `snapshot.storage.k8s.io` API (VolumeSnapshot / VolumeSnapshotContent / VolumeSnapshotClass) +# and the snapshot-controller that reconciles them are NOT part of k3s or of Longhorn — they are +# a SEPARATE, cluster-singleton component (kubernetes-csi/external-snapshotter). Without it the +# CRDs are absent (only `snapshots.longhorn.io` + the kubevirt.io snapshot CRDs exist), so: +# * no VolumeSnapshotClass can be created (item 2), and +# * CDI cannot do SNAPSHOT smart-clone (item 3) — it falls back to slow host-assisted `copy`, +# which leaves the default virt StorageClass "degraded" (CDIDefaultStorageClassDegraded). +# Installing it here (controllers pass) registers the CRDs; the VolumeSnapshotClass + the CDI +# StorageProfiles that CONSUME them are applied by platform-config (dependsOn: platform). +# +# CONTROLLERS/CONFIGS SPLIT (F-MON1): this is the CONTROLLERS half. The snapshot.storage.k8s.io +# CRDs are registered by the `client/config/crd` base below; the snapshot-controller Deployment +# (into kube-system, upstream default) reconciles VolumeSnapshots. Custom resources that use these +# CRDs (the Longhorn VolumeSnapshotClass, item 2) live in platform-config so the CRDs exist first. +# +# VERSION (Renovate-maintained): external-snapshotter v8.2.0. +# * The v8.x line updated its Kubernetes deps to 1.31 at v8.1.0 (this cluster is k3s v1.31.5), +# so v8.2.x is the conservative, version-matched choice — newer than the 1.31 alignment, +# older than the v8.4/v8.5 churn. +# * snapshot.storage.k8s.io CRDs are served at v1 (GA) — driver.longhorn.io speaks v1, so +# Longhorn CSI snapshots + CDI snapshot-clone work against these CRDs. +# * The upstream setup manifest pins the controller image to an older v8.0.x tag; the `images` +# transformer below re-pins it to v8.2.0 so the running controller matches the CRD version. +# * The `client/config/crd` base also ships the beta VolumeGroupSnapshot CRDs; that feature is +# gated OFF in the controller (no --enable-volume-group-snapshots arg), so they are inert. +# +# Kustomize loads these as REMOTE bases (github.com/...?ref=) — the same remote-manifest +# pattern platform/kubevirt uses for the KubeVirt/CDI operators; Flux's kustomize-controller +# fetches them identically at reconcile. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + # renovate: datasource=github-releases depName=kubernetes-csi/external-snapshotter + - https://github.com/kubernetes-csi/external-snapshotter/client/config/crd?ref=v8.2.0 + # renovate: datasource=github-releases depName=kubernetes-csi/external-snapshotter + - https://github.com/kubernetes-csi/external-snapshotter/deploy/kubernetes/snapshot-controller?ref=v8.2.0 +images: + # Re-pin the controller image to the base ref so controller == CRD version (see header note). + - name: registry.k8s.io/sig-storage/snapshot-controller + newTag: v8.2.0 diff --git a/platform/kubevirt/cdi-storageprofiles.yaml b/platform/kubevirt/cdi-storageprofiles.yaml new file mode 100644 index 0000000..f9eed35 --- /dev/null +++ b/platform/kubevirt/cdi-storageprofiles.yaml @@ -0,0 +1,46 @@ +--- +# CDI StorageProfiles — the storage "recipes" CDI uses when it provisions/clones VM disk PVCs +# (ADR-0005). CONSUME the cdi.kubevirt.io CRD registered by the CDI operator (platform pass), so — +# like cdi-cr.yaml — these CRs are applied by the `platform-config` Flux Kustomization +# (dependsOn: platform). They live here beside the KubeVirt/CDI component and are referenced in +# place from platform-config/kustomization.yaml. +# +# CDI AUTO-CREATES a StorageProfile per StorageClass and fills its STATUS (inferred capabilities). +# The operator owns `.status`; the USER owns `.spec`. Setting `.spec` here does NOT fight CDI — it +# overrides only the fields we set (server-side apply), and CDI keeps reconciling status around them. +# +# ── longhorn: turn on SNAPSHOT smart-clone ───────────────────────────────────────────────────── +# Default (empty spec) the longhorn profile reports `cloneStrategy: copy` — CDI clones a golden image +# by spinning a host-assisted byte-for-byte copy pod (slow, doubles I/O). With external-snapshotter +# installed and a driver.longhorn.io VolumeSnapshotClass present (../longhorn/volumesnapshotclass.yaml), +# `cloneStrategy: snapshot` makes CDI clone by taking a CSI VolumeSnapshot and provisioning the new +# PVC from it — near-instant, space-efficient. claimPropertySets left unset: CDI keeps inferring the +# class's native RWO/Filesystem, correct for ordinary (non-migratable) longhorn PVCs. +apiVersion: cdi.kubevirt.io/v1beta1 +kind: StorageProfile +metadata: + name: longhorn + annotations: + homelab.keithjackson.dev/adr: ADR-0005 +spec: + cloneStrategy: snapshot +--- +# ── longhorn-vm: SNAPSHOT smart-clone + RWX/Block for live-migratable VM disks ────────────────── +# This is the default *virtualization* StorageClass (is-default-virt-class in ../longhorn/ +# storageclasses.yaml). Its profile must advertise the access/volume mode combo Longhorn 1.7.3 +# supports for migratable RWX — ReadWriteMany + Block — so every VM DataVolume CDI provisions on this +# class is stamped RWX/Block and is therefore live-migratable. Together with cloneStrategy: snapshot +# (smart clone) this profile is NON-DEGRADED (supports smart-clone AND RWX), which is what clears the +# CDIDefaultStorageClassDegraded alert (the default virt class must support BOTH). +apiVersion: cdi.kubevirt.io/v1beta1 +kind: StorageProfile +metadata: + name: longhorn-vm + annotations: + homelab.keithjackson.dev/adr: ADR-0005 +spec: + cloneStrategy: snapshot + claimPropertySets: + - accessModes: + - ReadWriteMany + volumeMode: Block diff --git a/platform/kubevirt/kubevirt-cr.yaml b/platform/kubevirt/kubevirt-cr.yaml index 23fb3be..430031e 100644 --- a/platform/kubevirt/kubevirt-cr.yaml +++ b/platform/kubevirt/kubevirt-cr.yaml @@ -30,8 +30,18 @@ metadata: annotations: homelab.keithjackson.dev/adr: ADR-0005 spec: - # Drain VMs off a cordoned node when possible; software-emulated dev VMs just Evict+restart. - # (Live migration across the 3 converged nodes is a nice-to-have per docs/06, not required.) + # LIVE MIGRATION (docs/06): GA in KubeVirt since v0.43 — on v1.9 (this install) it is enabled by + # default and needs NO feature gate. developerConfiguration.featureGates is intentionally left + # EMPTY: the `LiveMigration` gate no longer exists as an alpha/beta toggle, and adding it would be + # a no-op (worse, an unknown gate name can be rejected). No spec.configuration.migrations tuning is + # added either — the defaults (parallelMigrationsPerCluster=5, parallelOutboundMigrationsPerNode=2, + # 64Mi/s bandwidth cap) are correct for a 3-node dev cluster; revisit only under WS-VM load tests. + # + # A VM live-migrates on node drain ONLY if it opts in per-VM with `evictionStrategy: LiveMigrate` + # AND its disks are on a RWX-migratable StorageClass — `longhorn-vm` (platform/longhorn/ + # storageclasses.yaml: migratable=true, RWX/Block). We do NOT set a cluster-wide + # spec.configuration.evictionStrategy: a blanket LiveMigrate would WEDGE drains for VMs whose disks + # are on the RWO `longhorn` class (unmigratable). VMs without opt-in still Evict+restart on drain. configuration: developerConfiguration: # Use real (nested) KVM. `false` makes virt-controller REQUEST the kvm device so /dev/kvm is diff --git a/platform/kustomization.yaml b/platform/kustomization.yaml index c13ca2d..24f9afd 100644 --- a/platform/kustomization.yaml +++ b/platform/kustomization.yaml @@ -14,6 +14,7 @@ resources: - kyverno # admission enforcer FIRST — policies present before other components reconcile - cilium # CNI (adopts the running Ansible-installed release; parity verified) - longhorn # replicated block storage / CSI + - external-snapshotter # CSI snapshot CRDs + snapshot-controller (VolumeSnapshot API; CDI smart-clone) - cert-manager # Let's Encrypt DNS-01 wildcard — needs SOPS `cloudflare-api-token` - externaldns # auto DNS records — needs SOPS `cloudflare-api-token` - gateway # Gateway API listeners (ingress; replaces Traefik) diff --git a/platform/longhorn/storageclasses.yaml b/platform/longhorn/storageclasses.yaml index d2c4446..49632d0 100644 --- a/platform/longhorn/storageclasses.yaml +++ b/platform/longhorn/storageclasses.yaml @@ -29,6 +29,52 @@ parameters: dataLocality: "disabled" # keep replicas spread for durability, not co-located with the pod fsType: "ext4" --- +# KubeVirt VM-disk tier — RWX + migratable, for LIVE-MIGRATABLE VM disks (ADR-0005, docs/06). +# +# WHY a second Longhorn class: the default `longhorn` class is ReadWriteOnce (a Longhorn volume +# attaches to ONE node). KubeVirt live migration needs the SOURCE and TARGET virt-launcher pods to +# hold the SAME disk on TWO nodes for the migration window, which requires ReadWriteMany. Longhorn +# provides this via the StorageClass parameter `migratable: "true"`. +# +# migratable RWX on Longhorn 1.7.3 is BLOCK-mode only (verified against Longhorn 1.7 docs + the live +# cluster: longhornio/longhorn-manager:v1.7.3). Longhorn's ORDINARY RWX (migratable=false) is an +# NFS/share-manager-backed *Filesystem* export; the MIGRATABLE RWX path is different — it is a raw +# block device that Longhorn allows attached to two nodes ONLY during a live migration (concurrent +# writers are not permitted, so a mounted filesystem could be corrupted). Therefore: +# * StorageClass parameter migratable: "true" +# * PVC/DataVolume accessModes ReadWriteMany +# * PVC/DataVolume volumeMode Block (Filesystem is NOT supported with migratable) +# The volumeMode/accessMode live on the PVC/DataVolume; CDI stamps them automatically for disks on +# this class via the matching StorageProfile claimPropertySets (../kubevirt/cdi-storageprofiles.yaml). +# +# is-default-virt-class: CDI treats THIS as the default *virtualization* StorageClass (precedence over +# the k8s default `longhorn` for VM disks when storageClassName is omitted). Because it supports smart +# clone (snapshot) AND ReadWriteMany, its StorageProfile is non-degraded — which is what clears the +# CDIDefaultStorageClassDegraded alert. It is NOT the cluster (k8s) default: `longhorn` keeps the +# storageclass.kubernetes.io/is-default-class annotation, so ordinary PVCs still land on RWO longhorn. +# Exactly one SC may carry is-default-virt-class (else CDIMultipleDefaultVirtStorageClasses fires). +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: longhorn-vm + annotations: + storageclass.kubevirt.io/is-default-virt-class: "true" +provisioner: driver.longhorn.io +allowVolumeExpansion: true +reclaimPolicy: Delete # dev: reclaim space when the PVC is deleted +volumeBindingMode: Immediate +parameters: + # Same 3-replica durability as the default class (HARD node anti-affinity in helmrelease.yaml). + # Quoted string literals — see the numberOfReplicas note on the `longhorn` class above. + numberOfReplicas: "3" + staleReplicaTimeout: "30" + dataLocality: "disabled" + # THE knob that makes this class usable for KubeVirt live migration: mark volumes migratable so a + # RWX Longhorn volume can be attached to two nodes during a live migration. (Block-mode only.) + migratable: "true" + # No fsType: migratable RWX volumes are Block-mode (raw device); a filesystem is never laid down + # by Longhorn here — the guest owns the block device. +--- # Bulk media tier — large, NON-replicated data (Plex library, download staging, arr handoff). # docs/04 / R-STATE-04: 3x replicating multi-TB media across 3 mini-PCs is impractical, so bulk # media lives OUTSIDE the cluster and is consumed as an external NFS/SMB mount via this thin diff --git a/platform/longhorn/volumesnapshotclass.yaml b/platform/longhorn/volumesnapshotclass.yaml new file mode 100644 index 0000000..6e1bbc5 --- /dev/null +++ b/platform/longhorn/volumesnapshotclass.yaml @@ -0,0 +1,28 @@ +--- +# Longhorn VolumeSnapshotClass — the CSI snapshot recipe for driver.longhorn.io. +# +# CONSUMES the snapshot.storage.k8s.io CRD registered by platform/external-snapshotter (controllers +# pass), so — like the KubeVirt/CDI CRs — this CR is applied by the `platform-config` Flux +# Kustomization (dependsOn: platform), not in the controllers pass. It lives HERE beside the Longhorn +# component for locality and is referenced in place from platform-config/kustomization.yaml. +# +# WHY: this is what turns "Longhorn has a CSI driver" into "the cluster can take CSI VolumeSnapshots +# of Longhorn PVCs". CDI's snapshot smart-clone (cdi-storageprofiles.yaml -> cloneStrategy: snapshot) +# resolves the driver's VolumeSnapshotClass by matching driver name; this is the only one for +# driver.longhorn.io, so CDI selects it unambiguously (no is-default-class annotation needed). +# +# type: snap -> an in-cluster Longhorn snapshot on the volume's own replicas (fast, no external +# backupstore required). The alternative `type: bak` uploads to a backup target, which +# is not configured until Phase 4 (Velero/backupTarget). Clone/restore stay in-cluster. +# deletionPolicy: Delete -> when the VolumeSnapshot is deleted, its Longhorn snapshot is cleaned up. +# CDI's clone snapshots are transient, so leaking them would waste replica space. +apiVersion: snapshot.storage.k8s.io/v1 +kind: VolumeSnapshotClass +metadata: + name: longhorn + labels: + app.kubernetes.io/part-of: longhorn +driver: driver.longhorn.io +deletionPolicy: Delete +parameters: + type: snap