jakman85andClaude Opus 4.8
d60e5531b1
The three KubeVirt VM conformance tests (vm-lifecycle, vm-default-deny,
vm-data-survival) carried comments asserting the ALPHA `ImageVolume` feature
gate is DISABLED "on this k3s v1.31 cluster" and that the apiserver therefore
prunes the container-disk `image` volume, breaking the containerDisk boot path.
The cluster was upgraded v1.31 -> v1.36.4. Revalidated the gate live:
- kubernetes_feature_enabled{name="ImageVolume"} = 1 (enabled by default;
stage="" — no longer ALPHA)
- a server-side `kubectl apply --dry-run=server` of a Pod with an `image`
volume RETAINS the volume + mount instead of pruning them.
So the old premise is stale: containerDisk roots would boot on 1.36.4. The
tests never asserted the gate state and never used containerDisk — they boot
from CDI DataVolume roots, which remains the correct choice because it is the
spec-required, rebuild-surviving disk model (docs/06, R-STATE-01); an ephemeral
containerDisk could never satisfy data-survival. Test logic is therefore
unchanged; only the explanatory comments are corrected (v1.31 -> v1.36.4, and
the "containerDisk is broken here" rationale marked stale with the live
evidence).
Merge gate: PASS (lint, l0, render-golden, l1 green).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
224 lines
11 KiB
YAML
224 lines
11 KiB
YAML
# =============================================================================
|
|
# L3 FAULT-INJECTION — VM disk survives rebuild (R-STATE-01) [RUNNABLE / self-skips]
|
|
# *SAFETY-CRITICAL*
|
|
# Catalog: docs/14 "Data survival (R-STATE-01, L3)" — the IaaS partner of the
|
|
# container `data-survival` test. Spec: docs/04, docs/06, ADR-0005.
|
|
#
|
|
# GIVEN a VM that writes data to a Longhorn-backed disk,
|
|
# WHEN the VM is DESTROYED entirely (kept: the Longhorn PVC/DataVolume),
|
|
# THEN the data is intact and readable by a fresh workload bound to that PVC —
|
|
# exactly the container proof (data-survival), now for a KubeVirt VM
|
|
# (R-STATE-01 covers PaaS + IaaS).
|
|
#
|
|
# METHOD (virtctl-free, filesystem-agnostic, deterministic): a VM whose ROOT is a
|
|
# CDI DataVolume on Longhorn (a mounted PVC — NOT an ephemeral containerDisk). A
|
|
# containerDisk root is EPHEMERAL — discarded when the VM is destroyed — so it could
|
|
# never prove data-survival; the spec-required disk model here is a Longhorn-backed
|
|
# DataVolume that OUTLIVES the VM (docs/06/R-STATE-01), which is exactly what this test
|
|
# destroys-and-rebuilds. (Feature-gate note, revalidated 2026-08: an earlier revision
|
|
# of this comment also claimed the containerDisk BOOT path was broken on this cluster —
|
|
# KubeVirt v1.9 serves the container-disk binary through a k8s `image` volume gated by
|
|
# the ALPHA `ImageVolume` feature gate, which k3s v1.31 had DISABLED, so the apiserver
|
|
# pruned the volume and the guest died at OCI init. That premise is now STALE: the
|
|
# cluster is k3s v1.36.4, where ImageVolume is ENABLED by default
|
|
# (kubernetes_feature_enabled{name="ImageVolume"}=1; a server-side dry-run RETAINS an
|
|
# `image` volume rather than pruning it), so containerDisk roots would boot — DataVolume
|
|
# is used purely because only it survives rebuild.) The VM also has a SECOND
|
|
# Longhorn BLOCK data disk; cloud-init writes a unique canary to that raw block device
|
|
# and powers the guest off; runStrategy RerunOnFailure means
|
|
# a clean shutdown STAYS down (VMI -> Succeeded), so the write is known-complete and
|
|
# the disk is released. We then DELETE the VM (keeping the PVC) and bind a throwaway
|
|
# busybox reader pod to the SAME Longhorn PVC as a raw block device — if the bytes
|
|
# match, the data survived the VM's destruction. No SSH/console scraping, no guest
|
|
# filesystem parsing.
|
|
#
|
|
# RUNNABLE / SELF-SKIPPING: skips cleanly (green) until BOTH KubeVirt and a
|
|
# `longhorn` StorageClass exist; runs the full destroy/rebuild survival check once
|
|
# they do. Hermetic: everything lives in the chainsaw per-test namespace.
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: vm-data-survival
|
|
labels:
|
|
conformance.home.k3s/R-STATE-01: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-STATE-01"
|
|
conformance.home.k3s/layer: "L3"
|
|
conformance.home.k3s/safety-critical: "true"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
Destroy a VM entirely, keep its Longhorn PVC, and a fresh workload re-bound to
|
|
that PVC still reads the VM-written data. Skips cleanly until KubeVirt + longhorn exist.
|
|
spec:
|
|
# CDI import onto Longhorn + boot a guest + run cloud-init + power off, then reattach a
|
|
# Longhorn RWO volume — far beyond 180s.
|
|
timeouts:
|
|
exec: 30m
|
|
steps:
|
|
- name: vm-data-survives-destroy-and-rebuild
|
|
try:
|
|
- script:
|
|
content: |
|
|
set -euo pipefail
|
|
NS="$NAMESPACE"
|
|
|
|
# -- Gate: skip cleanly (green) until BOTH KubeVirt and longhorn are present. --
|
|
if ! kubectl get crd virtualmachineinstances.kubevirt.io >/dev/null 2>&1; then
|
|
echo "SKIP: KubeVirt CRDs absent — vm-data-survival not applicable yet"
|
|
echo VM_DATA_SURVIVAL_SKIPPED_OK; exit 0
|
|
fi
|
|
if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
|
|
echo "SKIP: 'longhorn' StorageClass absent — vm-data-survival not applicable yet"
|
|
echo VM_DATA_SURVIVAL_SKIPPED_OK; exit 0
|
|
fi
|
|
|
|
CANARY="vm-canary-$(date +%s%N)" # fixed-length, unique per run
|
|
LEN=${#CANARY}
|
|
|
|
# -- GIVEN: a Longhorn BLOCK PVC for the VM's data disk. --
|
|
kubectl -n "$NS" apply -f - <<EOF
|
|
apiVersion: v1
|
|
kind: PersistentVolumeClaim
|
|
metadata:
|
|
name: vm-survivor
|
|
spec:
|
|
accessModes: [ReadWriteOnce]
|
|
volumeMode: Block
|
|
storageClassName: longhorn
|
|
resources:
|
|
requests:
|
|
storage: 1Gi
|
|
EOF
|
|
|
|
# -- The VM's ROOT disk: CDI imports Fedora onto a Longhorn PVC (bypasses the broken
|
|
# containerDisk/image-volume path; standalone DataVolume). Wait for the import. --
|
|
kubectl -n "$NS" apply -f - <<'EOF'
|
|
apiVersion: cdi.kubevirt.io/v1beta1
|
|
kind: DataVolume
|
|
metadata:
|
|
name: survivor-rootdisk
|
|
spec:
|
|
source:
|
|
registry:
|
|
url: "docker://quay.io/containerdisks/fedora:41"
|
|
storage:
|
|
accessModes:
|
|
- ReadWriteOnce
|
|
storageClassName: longhorn
|
|
resources:
|
|
requests:
|
|
storage: 6Gi
|
|
EOF
|
|
dv_phase=""
|
|
for i in $(seq 1 120); do
|
|
dv_phase="$(kubectl -n "$NS" get dv survivor-rootdisk -o jsonpath='{.status.phase}' 2>/dev/null || true)"
|
|
[ "$dv_phase" = "Succeeded" ] && break
|
|
sleep 10
|
|
done
|
|
[ "$dv_phase" = "Succeeded" ] || { echo "FAIL: CDI import did not Succeed (last='$dv_phase')" >&2; exit 1; }
|
|
echo "CDI import complete: survivor-rootdisk Succeeded (Longhorn PVC)"
|
|
|
|
# -- The VM: DataVolume root (Longhorn) + the Longhorn block disk (/dev/vdb) + a
|
|
# cloud-init disk. cloud-init writes the canary to the raw device and powers off.
|
|
# RerunOnFailure => a clean poweroff STAYS Succeeded (not restarted), so the write is
|
|
# known-complete and the disk released. Only $CANARY expands (no other '$' in the body).
|
|
kubectl -n "$NS" apply -f - <<EOF
|
|
apiVersion: kubevirt.io/v1
|
|
kind: VirtualMachine
|
|
metadata:
|
|
name: survivor
|
|
spec:
|
|
runStrategy: RerunOnFailure
|
|
template:
|
|
metadata:
|
|
labels:
|
|
kubevirt.io/vm: survivor
|
|
homelab.keithjackson.dev/egress-class: internet
|
|
spec:
|
|
domain:
|
|
# The containerdisks/fedora:41 image is UEFI-native (ESP + /boot mounted via
|
|
# systemd GPT auto-discovery). KubeVirt defaults to SeaBIOS, under which the guest
|
|
# boots to emergency mode (ESP + /boot by-uuid mounts time out) and cloud-init's
|
|
# runcmd/poweroff never runs. Boot with EFI (OVMF). secureBoot:false => no SMM needed.
|
|
firmware:
|
|
bootloader:
|
|
efi:
|
|
secureBoot: false
|
|
cpu: { cores: 1 }
|
|
memory: { guest: 1024Mi }
|
|
devices:
|
|
disks:
|
|
- name: rootdisk
|
|
disk: { bus: virtio }
|
|
- name: datadisk
|
|
disk: { bus: virtio }
|
|
- name: cloudinit
|
|
disk: { bus: virtio }
|
|
resources:
|
|
requests:
|
|
memory: 1024Mi
|
|
volumes:
|
|
- name: rootdisk
|
|
dataVolume:
|
|
name: survivor-rootdisk
|
|
- name: datadisk
|
|
persistentVolumeClaim:
|
|
claimName: vm-survivor
|
|
- name: cloudinit
|
|
cloudInitNoCloud:
|
|
userData: |-
|
|
#cloud-config
|
|
runcmd:
|
|
- [ sh, -c, "printf '%s' '$CANARY' | dd of=/dev/vdb bs=1 seek=0 conv=notrunc; sync; sleep 3; poweroff" ]
|
|
EOF
|
|
|
|
# -- WHEN: wait until the guest has written the canary and powered off (VMI Succeeded/gone). --
|
|
ok=""
|
|
for i in $(seq 1 90); do
|
|
phase="$(kubectl -n "$NS" get vmi survivor -o jsonpath='{.status.phase}' 2>/dev/null || true)"
|
|
if [ "$phase" = "Succeeded" ] || { [ -z "$phase" ] && [ "$i" -gt 3 ]; }; then ok=1; break; fi
|
|
sleep 10
|
|
done
|
|
[ -n "$ok" ] || { echo "FAIL: guest never completed the write+poweroff (last phase='$phase')" >&2; exit 1; }
|
|
echo "guest wrote canary and powered off"
|
|
|
|
# -- FAULT INJECTION: destroy the VM ENTIRELY; KEEP the Longhorn PVC. --
|
|
kubectl -n "$NS" delete vm survivor --wait=true --timeout=180s
|
|
kubectl -n "$NS" get pvc vm-survivor # must still exist
|
|
|
|
# -- THEN: a fresh workload re-bound to the SAME Longhorn PVC still sees the bytes. --
|
|
kubectl -n "$NS" apply -f - <<EOF
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata:
|
|
name: reader
|
|
spec:
|
|
restartPolicy: Never
|
|
containers:
|
|
- name: app
|
|
image: busybox:1.36.1
|
|
command: ["sh","-c","sleep 3600"]
|
|
volumeDevices:
|
|
- name: data
|
|
devicePath: /dev/datadisk
|
|
volumes:
|
|
- name: data
|
|
persistentVolumeClaim:
|
|
claimName: vm-survivor
|
|
EOF
|
|
kubectl -n "$NS" wait --for=condition=Ready pod/reader --timeout=240s
|
|
GOT="$(kubectl -n "$NS" exec reader -- sh -c "dd if=/dev/datadisk bs=1 count=$LEN 2>/dev/null")"
|
|
echo "recovered='$GOT' expected='$CANARY'"
|
|
[ "$GOT" = "$CANARY" ] && echo VM_DATA_SURVIVED_OK
|
|
check:
|
|
(contains($stdout, 'VM_DATA_SURVIVED_OK') || contains($stdout, 'VM_DATA_SURVIVAL_SKIPPED_OK')): true
|
|
finally:
|
|
- script:
|
|
content: |
|
|
kubectl -n "$NAMESPACE" delete vm survivor --ignore-not-found --wait=false || true
|
|
kubectl -n "$NAMESPACE" delete pod reader --ignore-not-found --wait=false || true
|
|
kubectl -n "$NAMESPACE" delete dv survivor-rootdisk --ignore-not-found --wait=false || true
|
|
kubectl -n "$NAMESPACE" delete pvc vm-survivor --ignore-not-found --wait=false || true
|
|
echo "cleanup best-effort; chainsaw namespace teardown finalizes"
|