fix(gitops): CDI cold-start deadlock repaired IN THE GRAPH, not by a pre-apply

`platform-config` applied the CDI CR *and* cdi-storageprofiles.yaml in ONE pass.
StorageProfile's CRD does not exist until cdi-operator reconciles that CR, and
Flux server-side dry-runs every object before applying any — so the unknown kind
aborted the tier, the CR was never applied, the CRD was never created, and the
tier could NEVER converge. Observed on prod: cdi-operator Available 12h, no CDI
CR, no storageprofiles CRD, every downstream tier blocked.

The profiles now live in their own tier ./platform-config-cdi with
dependsOn: platform-config and wait: true. Because dependsOn is REVISION-gated
and platform-config runs wait: true, the CDI CR must be applied AND healthy —
exactly the event that creates the CRD — before the profiles are applied. Cold
clusters order themselves. `standup.sh crds` is retained as a fallback for
clusters already stuck, and because it is idempotent.

THE GENERALISABLE RULE (recorded in clusters/README.md): a CR whose CRD is
created by a CONTROLLER AT RUNTIME is second-order and needs its own pass.
cdi-operator.yaml registers exactly ONE CRD (cdis.cdi.kubevirt.io); storageprofiles,
datavolumes, datasources and dataimportcrons appear only after it reconciles the
CDI CR. An audit found StorageProfile is the only second-order kind in
platform-config — KubeVirt, CDI, KeycloakRealmImport, CNPG Cluster,
VolumeSnapshotClass, ClusterIssuer, PrometheusRule, ServiceMonitor and the Cilium
L2 CRs all have their CRD in git or in a chart.

Moving the CDI CR up into `platform` was rejected: that would put CDI's health on
the tier the CNI gates and everything waits on — the mistake the oauth2-proxy and
headlamp splits exist to undo.

apps-vms gains dependsOn: platform-config-cdi. LOAD-BEARING, not cosmetic:
apps/vms/unifi/20-datavolume.yaml provisions on storageClassName: longhorn-vm and
its comment claimed platform-config guaranteed the profiles — which this move
would have silently falsified.

THE PRUNE TRANSIENT, MEASURED RATHER THAN ASSUMED. platform-config has
prune: true, so it deletes both profiles once, CDI recreates them with empty spec,
and the new tier re-applies the overrides ~30-60s later. Compared our override
against CDI's unaided inference for the same Longhorn provisioner on dev:
  longhorn-vm (we override)     spec RWX/Block  -> status ReadWriteMany/Block
  longhorn-static (no override) spec unset      -> status ReadWriteOnce/Block
Both infer cloneStrategy: snapshot. So cloneStrategy is NOT at risk; accessModes
IS — unaided CDI gives RWO. A DataVolume provisioned on longhorn-vm inside the
window would be stamped RWO/Block and be PERMANENTLY non-live-migratable. Flux can
no longer cause it (apps-vms gates on the new tier); a human must not do it by hand.
Corroborating evidence that the recreate happens at all: dev has 5 StorageClasses
and 5 StorageProfiles, three with fluxOwner=NONE, spec unset and status populated
— CDI creating and inferring them with no Flux involvement.

MERGE PRECONDITION: `flux get kustomization platform-config` must be Ready on the
target first. If it is not, the profiles are pruned and sit at CDI defaults until
it recovers — and prod was in that state hours earlier. Fallback is the two-step
the Cilium move used (merge with prune: false, verify the ownership label moved,
restore prune: true).

substituteFrom: cluster-vars is declared even though the manifest has no ${TOKEN},
deliberately: conformance/tests/l0-static/flux-postbuild-substitution only audits
paths whose Kustomization declares it, so omitting it would move the file OUT of
the audited set — a blind spot for a future literal-token ship. Gate confirms it
stayed in (path=./platform-config-cdi files=2 defined=0 hazards=0). No SOPS
decryption stanza: the moved file is two plain CRs, so no overlay change was needed.

VERIFIED INDEPENDENTLY OF THE AGENT: platform-config renders 0 StorageProfile
while keeping the CDI and KubeVirt CRs; platform-config-cdi renders exactly the
two profiles; the graph is byte-identical across dev and prod with apps-vms
carrying both edges and no cycle. Full CI gate PASS.

KNOWN FOLLOW-UP: bootstrap/standup.sh TIER_ORDER does not yet include
platform-config-cdi, so `resume` will not wait on it. Another agent holds that
file; the PM will add it.

Refs R-GITOPS-01, R-ENV-02.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-05 11:53:45 -05:00
1 parent 32d67660a6
commit 765ad032db
11 files changed
+330 -17

No files matched your search

+6 -2
View File
@@ -20,8 +20,12 @@ Neither k3s nor the Longhorn chart ships this. It is a prerequisite for:
## Wiring
- **Pass:** `platform` (controllers). Registers the CRDs.
- The `VolumeSnapshotClass` / `StorageProfile` CRs that consume the CRDs live in the
`platform-config` pass (`dependsOn: platform`), per the controllers/configs split.
- The `VolumeSnapshotClass` CR that consumes the CRDs lives in the `platform-config` pass
(`dependsOn: platform`), per the controllers/configs split.
- The CDI `StorageProfile`s live one pass further out, in `platform-config-cdi`
(`dependsOn: platform-config`) — `storageprofiles.cdi.kubevirt.io` is created by cdi-operator only
after it reconciles the `CDI` CR, so they cannot share a pass with it. See
`clusters/base/platform-config-cdi.yaml`.
## Version
@@ -9,8 +9,10 @@
# * no VolumeSnapshotClass can be created (item 2), and
# * CDI cannot do SNAPSHOT smart-clone (item 3) — it falls back to slow host-assisted `copy`,
# which leaves the default virt StorageClass "degraded" (CDIDefaultStorageClassDegraded).
# Installing it here (controllers pass) registers the CRDs; the VolumeSnapshotClass + the CDI
# StorageProfiles that CONSUME them are applied by platform-config (dependsOn: platform).
# Installing it here (controllers pass) registers the CRDs; the VolumeSnapshotClass that CONSUMES them
# is applied by platform-config (dependsOn: platform), and the CDI StorageProfiles that set
# `cloneStrategy: snapshot` by platform-config-cdi (dependsOn: platform-config — their own CRD is
# created by cdi-operator only after the `CDI` CR reconciles; see clusters/base/platform-config-cdi.yaml).
#
# CONTROLLERS/CONFIGS SPLIT (F-MON1): this is the CONTROLLERS half. The snapshot.storage.k8s.io
# CRDs are registered by the `client/config/crd` base below; the snapshot-controller Deployment
+14 -4
View File
@@ -1,9 +1,19 @@
---
# CDI StorageProfiles — the storage "recipes" CDI uses when it provisions/clones VM disk PVCs
# (ADR-0005). CONSUME the cdi.kubevirt.io CRD registered by the CDI operator (platform pass), so —
# like cdi-cr.yaml — these CRs are applied by the `platform-config` Flux Kustomization
# (dependsOn: platform). They live here beside the KubeVirt/CDI component and are referenced in
# place from platform-config/kustomization.yaml.
# (ADR-0005).
#
# APPLIED BY `platform-config-cdi`, NOT `platform-config` (moved 2026-09-05). These are SECOND-ORDER
# CRs: `storageprofiles.cdi.kubevirt.io` is NOT registered by the CDI operator install (cdi-operator.yaml
# ships only `cdis.cdi.kubevirt.io`) — cdi-operator creates it only AFTER it has reconciled the `CDI` CR
# that platform-config applies. Flux server-side dry-runs every object in a pass before applying any of
# them, so co-locating these with that CR was a TRUE DEADLOCK: unknown kind -> pass aborts -> CR never
# applied -> CRD never created -> platform-config never converges -> every tier below it blocked. It was
# invisible on dev (CDI CRDs long present) and cost 12h on prod's cold start, 2026-09-05.
# They live here beside the KubeVirt/CDI component and are referenced in place from
# platform-config-cdi/kustomization.yaml (dependsOn: platform-config, revision-gated).
#
# `apps-vms` dependsOn platform-config-cdi for these: a DataVolume created before the longhorn-vm
# profile below exists is stamped RWO/Filesystem and is permanently not live-migratable.
#
# CDI AUTO-CREATES a StorageProfile per StorageClass and fills its STATUS (inferred capabilities).
# The operator owns `.status`; the USER owns `.spec`. Setting `.spec` here does NOT fight CDI — it
+10
View File
@@ -8,6 +8,16 @@
# (dependsOn: platform, wait) so the CRDs exist BEFORE the CRs apply. Applying both in one atomic pass
# deadlocks ("no matches for kind KubeVirt"), exactly the failure F-MON1 documents.
#
# THERE IS A THIRD PASS FOR CDI, and it is not optional (2026-09-05). The two operator manifests below
# register exactly ONE CDI CRD, `cdis.cdi.kubevirt.io`. Every other CDI CRD — `storageprofiles`,
# `datavolumes`, `datasources`, `dataimportcrons` — is created by cdi-operator only AFTER it reconciles
# the `CDI` CR. So `cdi-storageprofiles.yaml` is a SECOND-ORDER CR and CANNOT share a pass with
# `cdi-cr.yaml`: the dry-run of the unknown `StorageProfile` kind aborts the pass, the CR is never
# applied, the CRD is never created, and platform-config deadlocks (cost 12h on prod's cold start).
# It is therefore applied by a third Kustomization, `platform-config-cdi` (dependsOn platform-config).
# Before adding any new CDI CR here, ask which pass owns it: CRD in git -> platform-config;
# CRD created by cdi-operator at runtime -> platform-config-cdi.
#
# Namespaces: the operator manifests each create their own namespace (`kubevirt`, `cdi`) with the
# pod-security profile they need (kubevirt is privileged) — so we do NOT redeclare them here (that
# would be a duplicate-resource kustomize error). These are PLATFORM namespaces: they intentionally