jakman85andClaude Opus 4.8
d60e5531b1
The three KubeVirt VM conformance tests (vm-lifecycle, vm-default-deny,
vm-data-survival) carried comments asserting the ALPHA `ImageVolume` feature
gate is DISABLED "on this k3s v1.31 cluster" and that the apiserver therefore
prunes the container-disk `image` volume, breaking the containerDisk boot path.
The cluster was upgraded v1.31 -> v1.36.4. Revalidated the gate live:
- kubernetes_feature_enabled{name="ImageVolume"} = 1 (enabled by default;
stage="" — no longer ALPHA)
- a server-side `kubectl apply --dry-run=server` of a Pod with an `image`
volume RETAINS the volume + mount instead of pruning them.
So the old premise is stale: containerDisk roots would boot on 1.36.4. The
tests never asserted the gate state and never used containerDisk — they boot
from CDI DataVolume roots, which remains the correct choice because it is the
spec-required, rebuild-surviving disk model (docs/06, R-STATE-01); an ephemeral
containerDisk could never satisfy data-survival. Test logic is therefore
unchanged; only the explanatory comments are corrected (v1.31 -> v1.36.4, and
the "containerDisk is broken here" rationale marked stale with the live
evidence).
Merge gate: PASS (lint, l0, render-golden, l1 green).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
244 lines
12 KiB
YAML
244 lines
12 KiB
YAML
# =============================================================================
|
|
# L2 BEHAVIORAL — Default-deny applies to a VM (R-NET-01) [RUNNABLE / self-skips]
|
|
# Catalog: docs/14 (default-deny) — the IaaS partner of the container `default-deny`
|
|
# test. Spec: docs/03 "Default-deny baseline"; docs/06 ("default-deny +
|
|
# egress classes apply to VMs too"); ADR-0005/0015.
|
|
#
|
|
# GIVEN a VM in a namespace carrying the default-deny baseline (deny all east-west +
|
|
# egress, allow only DNS to the in-cluster resolver),
|
|
# WHEN the guest probes the network from INSIDE the VM,
|
|
# THEN DNS resolves (the one standing allow) but an undeclared public TCP dial is
|
|
# BLOCKED — the same default-deny fabric that governs pods governs the VM's
|
|
# own datapath (its traffic egresses via the virt-launcher pod netns, so a
|
|
# podSelector:{} NetworkPolicy covers it). R-NET-01 for IaaS.
|
|
#
|
|
# METHOD (virtctl-free): the VM's ROOT is a CDI DataVolume on Longhorn (a mounted
|
|
# PVC — NOT an ephemeral containerDisk), the spec-required, rebuild-surviving disk model
|
|
# for VMs (docs/06, R-STATE-01), matching fixtures/vm.yaml. (Feature-gate note,
|
|
# revalidated 2026-08: an earlier revision said the containerDisk boot path was broken on
|
|
# this cluster — KubeVirt v1.9 serves the container-disk binary via a k8s `image` volume
|
|
# gated by the ALPHA `ImageVolume` feature gate, which k3s v1.31 had DISABLED, so the
|
|
# apiserver pruned it and the guest died at OCI init. That premise is now STALE: the
|
|
# cluster is k3s v1.36.4, where ImageVolume is ENABLED by default
|
|
# (kubernetes_feature_enabled{name="ImageVolume"}=1; a server-side dry-run RETAINS an
|
|
# `image` volume rather than pruning it), so containerDisk roots would boot — DataVolume
|
|
# is kept because it is the rebuild-surviving model.) cloud-init runs the two probes and writes a
|
|
# PASS/FAIL verdict to a raw Longhorn block disk, then powers off (RerunOnFailure =>
|
|
# stays down). A busybox reader pod reads the verdict back. The container default-deny
|
|
# test proves the datapath fail-closed (default-deny-failclosed, L3); this asserts
|
|
# the SAME baseline is enforced for a VM.
|
|
#
|
|
# RUNNABLE / SELF-SKIPPING: skips cleanly (green) until KubeVirt + longhorn exist.
|
|
#
|
|
# NOTE (bounded live-iteration, per the WS-VM report): the in-guest probe details
|
|
# (guest block-device ordering /dev/vdb, cluster-DNS injection into the guest, sh=bash
|
|
# for /dev/tcp on the Fedora containerDisk) are authored to the KubeVirt defaults and
|
|
# confirmed on the first live Phase-5 run; the test STRUCTURE and assertion are final.
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: vm-default-deny
|
|
labels:
|
|
conformance.home.k3s/R-NET-01: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-NET-01"
|
|
conformance.home.k3s/layer: "L2"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
A VM under the default-deny baseline reaches only DNS from inside the guest;
|
|
an undeclared public TCP dial is blocked. Skips cleanly until KubeVirt + longhorn exist.
|
|
spec:
|
|
timeouts:
|
|
exec: 30m
|
|
steps:
|
|
- name: default-deny-applies-to-the-vm-datapath
|
|
try:
|
|
- script:
|
|
content: |
|
|
set -euo pipefail
|
|
NS="$NAMESPACE"
|
|
|
|
# -- Gate: skip cleanly (green) until KubeVirt + longhorn exist. --
|
|
if ! kubectl get crd virtualmachineinstances.kubevirt.io >/dev/null 2>&1; then
|
|
echo "SKIP: KubeVirt CRDs absent"; echo VM_DEFAULT_DENY_SKIPPED_OK; exit 0
|
|
fi
|
|
if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
|
|
echo "SKIP: 'longhorn' StorageClass absent"; echo VM_DEFAULT_DENY_SKIPPED_OK; exit 0
|
|
fi
|
|
|
|
CANARY="vm-deny-$(date +%s%N)"
|
|
|
|
# -- GIVEN (disk): CDI imports the Fedora root onto a Longhorn PVC. This MUST happen
|
|
# BEFORE the default-deny NetworkPolicy below: default-deny (podSelector {}) also
|
|
# matches the CDI importer pod and would sever its egress to quay.io. The import is
|
|
# a one-time provisioning step; once the PVC holds the disk the VM boots with NO
|
|
# external egress, so applying default-deny AFTER the import still proves the VM's
|
|
# own datapath is fail-closed (the runbook's app-namespace equivalent is a scoped
|
|
# CiliumNetworkPolicy for the importer — see fixtures/vm.yaml). --
|
|
kubectl -n "$NS" apply -f - <<'EOF'
|
|
apiVersion: cdi.kubevirt.io/v1beta1
|
|
kind: DataVolume
|
|
metadata:
|
|
name: prober-rootdisk
|
|
spec:
|
|
source:
|
|
registry:
|
|
url: "docker://quay.io/containerdisks/fedora:41"
|
|
storage:
|
|
accessModes:
|
|
- ReadWriteOnce
|
|
storageClassName: longhorn
|
|
resources:
|
|
requests:
|
|
storage: 6Gi
|
|
EOF
|
|
dv_phase=""
|
|
for i in $(seq 1 120); do
|
|
dv_phase="$(kubectl -n "$NS" get dv prober-rootdisk -o jsonpath='{.status.phase}' 2>/dev/null || true)"
|
|
[ "$dv_phase" = "Succeeded" ] && break
|
|
sleep 10
|
|
done
|
|
[ "$dv_phase" = "Succeeded" ] || { echo "FAIL: CDI import did not Succeed (last='$dv_phase')" >&2; exit 1; }
|
|
echo "CDI import complete: prober-rootdisk Succeeded (Longhorn PVC)"
|
|
|
|
# -- GIVEN: the default-deny baseline in this namespace (deny all, allow only DNS to
|
|
# kube-system) — the exact shape Kyverno generates for app namespaces (docs/03).
|
|
# Applied AFTER the import so it governs the running VM's datapath (the point of the test). --
|
|
kubectl -n "$NS" apply -f - <<'EOF'
|
|
apiVersion: networking.k8s.io/v1
|
|
kind: NetworkPolicy
|
|
metadata:
|
|
name: default-deny
|
|
spec:
|
|
podSelector: {}
|
|
policyTypes: [Ingress, Egress]
|
|
ingress: []
|
|
egress:
|
|
- to:
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
kubernetes.io/metadata.name: kube-system
|
|
ports:
|
|
- { protocol: UDP, port: 53 }
|
|
- { protocol: TCP, port: 53 }
|
|
EOF
|
|
|
|
# -- The verdict block disk (Longhorn). --
|
|
kubectl -n "$NS" apply -f - <<'EOF'
|
|
apiVersion: v1
|
|
kind: PersistentVolumeClaim
|
|
metadata:
|
|
name: vm-verdict
|
|
spec:
|
|
accessModes: [ReadWriteOnce]
|
|
volumeMode: Block
|
|
storageClassName: longhorn
|
|
resources:
|
|
requests:
|
|
storage: 1Gi
|
|
EOF
|
|
|
|
# -- The VM + the in-guest probe (quoted heredoc: many '$' in the probe; __CANARY__ sed'd). --
|
|
cat > /tmp/vm-deny.yaml <<'EOF'
|
|
apiVersion: kubevirt.io/v1
|
|
kind: VirtualMachine
|
|
metadata:
|
|
name: prober
|
|
spec:
|
|
runStrategy: RerunOnFailure
|
|
template:
|
|
metadata:
|
|
labels:
|
|
kubevirt.io/vm: prober
|
|
homelab.keithjackson.dev/egress-class: internet
|
|
spec:
|
|
domain:
|
|
# The containerdisks/fedora:41 image is UEFI-native (ESP + /boot mounted via
|
|
# systemd GPT auto-discovery). KubeVirt defaults to SeaBIOS, under which the guest
|
|
# boots to emergency mode (ESP + /boot by-uuid mounts time out -> local-fs.target
|
|
# fails) and cloud-init's runcmd/poweroff never runs. Boot with EFI (OVMF) so the
|
|
# guest reaches multi-user and the probe executes. secureBoot:false => no SMM needed.
|
|
firmware:
|
|
bootloader:
|
|
efi:
|
|
secureBoot: false
|
|
cpu: { cores: 1 }
|
|
memory: { guest: 1024Mi }
|
|
devices:
|
|
disks:
|
|
- name: rootdisk
|
|
disk: { bus: virtio }
|
|
- name: verdict
|
|
disk: { bus: virtio }
|
|
- name: cloudinit
|
|
disk: { bus: virtio }
|
|
resources:
|
|
requests:
|
|
memory: 1024Mi
|
|
volumes:
|
|
- name: rootdisk
|
|
dataVolume:
|
|
name: prober-rootdisk
|
|
- name: verdict
|
|
persistentVolumeClaim:
|
|
claimName: vm-verdict
|
|
- name: cloudinit
|
|
cloudInitNoCloud:
|
|
userData: |-
|
|
#cloud-config
|
|
runcmd:
|
|
- [ sh, -c, "getent hosts quay.io >/dev/null 2>&1 && DNS=1 || DNS=0; timeout 5 sh -c 'echo > /dev/tcp/1.1.1.1/443' >/dev/null 2>&1 && DENY=0 || DENY=1; V=FAIL; [ \"$DNS\" = 1 ] && [ \"$DENY\" = 1 ] && V=PASS; printf '%s' \"$V:__CANARY__\" | dd of=/dev/vdb bs=1 seek=0 conv=notrunc; sync; sleep 3; poweroff" ]
|
|
EOF
|
|
sed -i "s/__CANARY__/$CANARY/" /tmp/vm-deny.yaml
|
|
kubectl -n "$NS" apply -f /tmp/vm-deny.yaml
|
|
|
|
# -- WHEN: wait until the guest has run the probes and powered off. --
|
|
ok=""
|
|
for i in $(seq 1 90); do
|
|
phase="$(kubectl -n "$NS" get vmi prober -o jsonpath='{.status.phase}' 2>/dev/null || true)"
|
|
if [ "$phase" = "Succeeded" ] || { [ -z "$phase" ] && [ "$i" -gt 3 ]; }; then ok=1; break; fi
|
|
sleep 10
|
|
done
|
|
[ -n "$ok" ] || { echo "FAIL: guest never completed the probes+poweroff (last='$phase')" >&2; exit 1; }
|
|
|
|
# -- THEN: read the verdict from the Longhorn disk via a throwaway reader. --
|
|
kubectl -n "$NS" delete vm prober --wait=true --timeout=180s
|
|
kubectl -n "$NS" apply -f - <<'EOF'
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata:
|
|
name: verdict-reader
|
|
spec:
|
|
restartPolicy: Never
|
|
containers:
|
|
- name: app
|
|
image: busybox:1.36.1
|
|
command: ["sh","-c","sleep 3600"]
|
|
volumeDevices:
|
|
- name: v
|
|
devicePath: /dev/verdict
|
|
volumes:
|
|
- name: v
|
|
persistentVolumeClaim:
|
|
claimName: vm-verdict
|
|
EOF
|
|
kubectl -n "$NS" wait --for=condition=Ready pod/verdict-reader --timeout=240s
|
|
GOT="$(kubectl -n "$NS" exec verdict-reader -- sh -c "dd if=/dev/verdict bs=1 count=$((5 + ${#CANARY})) 2>/dev/null")"
|
|
echo "verdict='$GOT' (expected PASS:$CANARY)"
|
|
case "$GOT" in
|
|
PASS:"$CANARY") echo VM_DEFAULT_DENY_OK ;;
|
|
*) echo "FAIL: default-deny not enforced for the VM (verdict='$GOT')" >&2; exit 1 ;;
|
|
esac
|
|
check:
|
|
(contains($stdout, 'VM_DEFAULT_DENY_OK') || contains($stdout, 'VM_DEFAULT_DENY_SKIPPED_OK')): true
|
|
finally:
|
|
- script:
|
|
content: |
|
|
kubectl -n "$NAMESPACE" delete vm prober --ignore-not-found --wait=false || true
|
|
kubectl -n "$NAMESPACE" delete pod verdict-reader --ignore-not-found --wait=false || true
|
|
kubectl -n "$NAMESPACE" delete dv prober-rootdisk --ignore-not-found --wait=false || true
|
|
kubectl -n "$NAMESPACE" delete pvc vm-verdict --ignore-not-found --wait=false || true
|
|
echo "cleanup best-effort; chainsaw namespace teardown finalizes"
|