home-network-k3s-impl/runbooks/cdi-import-stuck.md

# cdi-import-stuck — A CDI DataVolume import is not progressing

**Alert:** `CDIDataVolumeImportStuck` (warning, 30m)
**File:** `platform/kubevirt/alerts.yaml`, group `platform-health.kubevirt`
**Expression:** `sum(kubevirt_cdi_import_pods_high_restart) > 0`

## What this means here

A CDI DataVolume import or clone has not completed after 30 minutes, so **the VM disk it backs cannot
start**. `kubevirt_cdi_import_pods_high_restart` counts importer pods stuck restarting; a sustained
positive value means the import is not making progress rather than merely being slow.

The 30-minute `for` is generous on purpose: on this hardware a genuine large import is legitimately slow.
The reference case is the **UniFi VM's 60 GiB import**, and the `apps-vms` tier is deliberately
**non-gating** (`wait: false`, `dependsOn: platform-config`) precisely because a VM's image import plus
in-guest install is legitimately long-not-Ready.

**Two things to know before triaging:**

1. **Imports are the single heaviest write this platform performs**, and the etcd WAL (`/data/k3s`) shares
   one partition with every Longhorn replica (`/data/longhorn`). A 60 GiB import is exactly the class of
   write load behind the 2026-09-07 `k3s-prod-1` NotReady outage. So "restart the import" is not a free
   action — check `/runbooks/node-io` first.
2. **There is a known false-positive precedent in this area.** A broken HTTP CDI ServiceMonitor once caused
   a false `CDIOperatorDown` (commit `6a7e9d1`), and the sibling `KubeVirtControlPlaneDown` once used a
   non-existent metric with the wrong job label (`b57b3e9`). The metric name in this rule is flagged in
   the file as one to confirm against the deployed CDI. `cdi-prometheus-metrics` **is** a live scrape job
   on prod (measured 2026-09-13), so the target is up — but verify the series exists before trusting either
   this alert's firing or its silence.

Platform versions: **CDI v1.66.0** with **KubeVirt v1.9.0**, on real nested KVM.
## Running the commands in this runbook

`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.

## First commands

Which DataVolume, and how far it got:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get datavolumes.cdi.kubevirt.io -A \
     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PHASE:.status.phase,PROGRESS:.status.progress,RESTARTS:.status.restartCount'
```

The importer pod is where the real error is:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get pods -A | grep -E "importer|upload|clone"; echo; \
   kubectl --context k3s-prod -n unifi logs -l app=containerized-data-importer --tail=100 2>&1 | tail -50'
```

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n unifi describe pod -l app=containerized-data-importer 2>&1 | sed -n "/Events:/,\$p"'
```

Is there anywhere to put it — and is the scratch class configured?

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get pvc -A | grep -Ei "scratch|dv|rootdisk"; echo; \
   kubectl --context k3s-prod get sc; echo; \
   kubectl --context k3s-prod get cdis.cdi.kubevirt.io -o yaml | grep -A5 scratchSpace'
```

Is the node being flattened by the import? **Check this before restarting anything:**

```
ssh k3s-prod-1 'cat /proc/pressure/io; df -h /data'
```

Can the importer reach the image source at all? (Default-deny is the baseline everywhere.)

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \
     hubble observe --verdict DROPPED --last 100 -o compact | grep -i import'
```

Does the metric this alert depends on actually exist?

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   for q in "up{job=\"cdi-prometheus-metrics\"}" "count(kubevirt_cdi_import_pods_high_restart)"; do \
     printf "%-50s " "$q"; curl -s -G --data-urlencode "query=$q" localhost:19090/api/v1/query \
       | python3 -c "import json,sys;print([x[\"value\"][1] for x in json.load(sys.stdin)[\"data\"][\"result\"]] or \"ABSENT\")"; done'
```

## Likely causes, most likely first

### 1. Not enough schedulable storage for the target or the scratch space
The recorded failure, and the reason a whole StorageClass exists: **the 3-replica default class failed the
UniFi 60Gi import with insufficient storage / `ReplicaSchedulingFailure`.** The fix was a dedicated
single-replica **`longhorn-scratch`** class (`numberOfReplicas: 1`, `dataLocality: best-effort`),
referenced by the CDI CR's `scratchSpaceStorageClass`. If scratch is pointed at the 3-replica default,
this is your cause. Note that a 60 GiB import needing 3x replication plus scratch space is a very
different ask from 60 GiB.

### 2. An egress or default-deny block on the image source
Import pods are ordinary pods under the cluster's default-deny baseline (R-NET-01). The alert's own
description names this: *"often an egress/default-deny block on the image source, or a slow mirror."*
Remediation is an explicit allow in git; see `/runbooks/denied-flows` — and in particular the
post-Cilium-upgrade finding that `toEntities: world` no longer covers a destination that resolves to an
in-cluster backend.

### 3. The importer cannot open the raw Block device
`nonroot-devices: true` is set on the k3s servers because the **non-root CDI importer must open a raw
Block device that Kubernetes never chowns for it** (commit `df5979f`). A permission error in the importer
log against a Block-mode PVC is this. `longhorn-vm` is Block-mode only, so VM disks are always in scope.

### 4. The source is slow, unreachable, or the URL is wrong
A 404/403 or a stalled transfer appears plainly in the importer log. A slow mirror is a genuine cause — the
30m `for` already allows for it, so check throughput rather than assuming a fault.

### 5. Node I/O pressure is starving the import
Self-reinforcing: the import is heavy, the disk is shared with etcd, and a node under I/O pressure slows
the import which then runs longer. `/runbooks/node-io`.

### 6. Longhorn is degraded
An import onto a degraded volume competes with the rebuild. `/runbooks/longhorn-degraded`.

### 7. The alert is wrong
Given the CDI/KubeVirt false-positive precedents, if the DataVolumes all read `Succeeded` this is a
monitoring fault. Fixing the rule is a real change, not a shrug — a permanently-firing alert is a
recorded, expensive failure mode here.

## When to escalate or stop

- **Check node I/O pressure BEFORE restarting or recreating an import.** A 60 GiB re-import is hours of
  write load on the partition the embedded etcd fsyncs to, and that combination took a node out of the
  cluster on 2026-09-07. If I/O-PSI is already elevated, wait or freeze Longhorn rebuilds first
  (`/runbooks/longhorn-degraded`).
- **Do not delete the target DataVolume/PVC casually** — that guarantees a full re-import.
- **Do not point `scratchSpaceStorageClass` at the 3-replica default** to "keep things consistent". The
  single-replica class exists because the default one demonstrably failed.
- **Do not widen a NetworkPolicy to `world`** to let an import out; author the narrow allow.
- Scope it honestly: the VM tier is non-gating, so a stuck import does not affect the container platform.
  The exception is the UniFi VM, which is the home network's controller.
- Escalate for k3s server-flag changes (`nonroot-devices`) or CDI CR changes — Ansible and operator work.

---
Source of truth: `runbooks/cdi-import-stuck.md`. Index: `/runbooks/`
Records: `DECISIONS-LOG.md` (the UniFi 60Gi import failing on the 3-replica default; `longhorn-scratch`);
commits `df5979f` (`nonroot-devices`), `6a7e9d1` and `b57b3e9` (false CDI/KubeVirt alerts).