home-network-k3s-impl/runbooks/cdi-import-stuck.md

cdi-import-stuck — A CDI DataVolume import is not progressing

Alert: CDIDataVolumeImportStuck (warning, 30m) File: platform/kubevirt/alerts.yaml, group platform-health.kubevirt Expression: sum(kubevirt_cdi_import_pods_high_restart) > 0

What this means here

A CDI DataVolume import or clone has not completed after 30 minutes, so the VM disk it backs cannot start. kubevirt_cdi_import_pods_high_restart counts importer pods stuck restarting; a sustained positive value means the import is not making progress rather than merely being slow.

The 30-minute for is generous on purpose: on this hardware a genuine large import is legitimately slow. The reference case is the UniFi VM's 60 GiB import, and the apps-vms tier is deliberately non-gating (wait: false, dependsOn: platform-config) precisely because a VM's image import plus in-guest install is legitimately long-not-Ready.

Two things to know before triaging:

1. Imports are the single heaviest write this platform performs, and the etcd WAL (/data/k3s) shares one partition with every Longhorn replica (/data/longhorn). A 60 GiB import is exactly the class of write load behind the 2026-09-07 k3s-prod-1 NotReady outage. So "restart the import" is not a free action — check /runbooks/node-io first.
2. There is a known false-positive precedent in this area. A broken HTTP CDI ServiceMonitor once caused a false CDIOperatorDown (commit 6a7e9d1), and the sibling KubeVirtControlPlaneDown once used a non-existent metric with the wrong job label (b57b3e9). The metric name in this rule is flagged in the file as one to confirm against the deployed CDI. cdi-prometheus-metrics is a live scrape job on prod (measured 2026-09-13), so the target is up — but verify the series exists before trusting either this alert's firing or its silence.

Platform versions: CDI v1.66.0 with KubeVirt v1.9.0, on real nested KVM.

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two dev.sh invocations at once; they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git. main is a protected branch — a direct push is rejected. A change lands by pushing a branch, opening a PR, and merging it once the gate is green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

Which DataVolume, and how far it got:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get datavolumes.cdi.kubevirt.io -A \
     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PHASE:.status.phase,PROGRESS:.status.progress,RESTARTS:.status.restartCount'

The importer pod is where the real error is:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get pods -A | grep -E "importer|upload|clone"; echo; \
   kubectl --context k3s-prod -n unifi logs -l app=containerized-data-importer --tail=100 2>&1 | tail -50'

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n unifi describe pod -l app=containerized-data-importer 2>&1 | sed -n "/Events:/,\$p"'

Is there anywhere to put it — and is the scratch class configured?

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get pvc -A | grep -Ei "scratch|dv|rootdisk"; echo; \
   kubectl --context k3s-prod get sc; echo; \
   kubectl --context k3s-prod get cdis.cdi.kubevirt.io -o yaml | grep -A5 scratchSpace'

Is the node being flattened by the import? Check this before restarting anything:

ssh k3s-prod-1 'cat /proc/pressure/io; df -h /data'

Can the importer reach the image source at all? (Default-deny is the baseline everywhere.)

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \
     hubble observe --verdict DROPPED --last 100 -o compact | grep -i import'

Does the metric this alert depends on actually exist?

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   for q in "up{job=\"cdi-prometheus-metrics\"}" "count(kubevirt_cdi_import_pods_high_restart)"; do \
     printf "%-50s " "$q"; curl -s -G --data-urlencode "query=$q" localhost:19090/api/v1/query \
       | python3 -c "import json,sys;print([x[\"value\"][1] for x in json.load(sys.stdin)[\"data\"][\"result\"]] or \"ABSENT\")"; done'

Likely causes, most likely first

1. Not enough schedulable storage for the target or the scratch space

The recorded failure, and the reason a whole StorageClass exists: the 3-replica default class failed the UniFi 60Gi import with insufficient storage / ReplicaSchedulingFailure. The fix was a dedicated single-replica longhorn-scratch class (numberOfReplicas: 1, dataLocality: best-effort), referenced by the CDI CR's scratchSpaceStorageClass. If scratch is pointed at the 3-replica default, this is your cause. Note that a 60 GiB import needing 3x replication plus scratch space is a very different ask from 60 GiB.

2. An egress or default-deny block on the image source

Import pods are ordinary pods under the cluster's default-deny baseline (R-NET-01). The alert's own description names this: "often an egress/default-deny block on the image source, or a slow mirror." Remediation is an explicit allow in git; see /runbooks/denied-flows — and in particular the post-Cilium-upgrade finding that toEntities: world no longer covers a destination that resolves to an in-cluster backend.

3. The importer cannot open the raw Block device

nonroot-devices: true is set on the k3s servers because the non-root CDI importer must open a raw Block device that Kubernetes never chowns for it (commit df5979f). A permission error in the importer log against a Block-mode PVC is this. longhorn-vm is Block-mode only, so VM disks are always in scope.

4. The source is slow, unreachable, or the URL is wrong

A 404/403 or a stalled transfer appears plainly in the importer log. A slow mirror is a genuine cause — the 30m for already allows for it, so check throughput rather than assuming a fault.

5. Node I/O pressure is starving the import

Self-reinforcing: the import is heavy, the disk is shared with etcd, and a node under I/O pressure slows the import which then runs longer. /runbooks/node-io.

6. Longhorn is degraded

An import onto a degraded volume competes with the rebuild. /runbooks/longhorn-degraded.

7. The alert is wrong

Given the CDI/KubeVirt false-positive precedents, if the DataVolumes all read Succeeded this is a monitoring fault. Fixing the rule is a real change, not a shrug — a permanently-firing alert is a recorded, expensive failure mode here.

When to escalate or stop

• Check node I/O pressure BEFORE restarting or recreating an import. A 60 GiB re-import is hours of write load on the partition the embedded etcd fsyncs to, and that combination took a node out of the cluster on 2026-09-07. If I/O-PSI is already elevated, wait or freeze Longhorn rebuilds first (/runbooks/longhorn-degraded).
• Do not delete the target DataVolume/PVC casually — that guarantees a full re-import.
• Do not point scratchSpaceStorageClass at the 3-replica default to "keep things consistent". The single-replica class exists because the default one demonstrably failed.
• Do not widen a NetworkPolicy to world to let an import out; author the narrow allow.
• Scope it honestly: the VM tier is non-gating, so a stuck import does not affect the container platform. The exception is the UniFi VM, which is the home network's controller.
• Escalate for k3s server-flag changes (nonroot-devices) or CDI CR changes — Ansible and operator work.

---

Source of truth: runbooks/cdi-import-stuck.md. Index: /runbooks/ Records: DECISIONS-LOG.md (the UniFi 60Gi import failing on the 3-replica default; longhorn-scratch); commits df5979f (nonroot-devices), 6a7e9d1 and b57b3e9 (false CDI/KubeVirt alerts).