--- # Flux Kustomization: SECOND-ORDER CDI CONFIGS — reconciles ./platform-config-cdi. # Serves R-GITOPS-01 (all desired state in git, reconciled by Flux) and R-ENV-02. # # WHY THIS TIER EXISTS — the durable fix for the 2026-09-05 prod CDI DEADLOCK # # `platform-config` applied the `CDI` CR and the CDI `StorageProfile`s in the SAME pass. # kustomize-controller server-side DRY-RUNS EVERY object before applying ANY of them, and # `storageprofiles.cdi.kubevirt.io` does not exist until cdi-operator has reconciled that `CDI` CR. # So the unknown kind aborted the whole tier → the CR was never applied → the CRD was never created → # platform-config could NEVER converge and every tier below it stayed blocked. Not a race: a # deadlock. cdi-operator was Available for 12h on prod with no CR. Invisible on dev (its CDI CRDs # have existed for months), FATAL on any cold cluster. See DECISIONS-LOG 2026-09-05. # # `bootstrap/standup.sh crds` pre-applies the `CDI` CR to break it by hand. That is a WORKAROUND and # stays as belt-and-braces for clusters already wedged; THIS Kustomization is the fix that makes the # graph order itself, so a cold cluster needs no human in the loop. # # THE ORDERING GUARANTEE, precisely: `dependsOn` is REVISION-gated, not merely Ready-gated (verified # against kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies`, our # version): sharing a GitRepository with the dependency, this Kustomization cannot reconcile revision # N until `platform-config` has APPLIED revision N — and, because platform-config runs `wait: true`, # not until the `CDI` CR it applied is HEALTHY. cdi-operator creates the remaining CDI CRDs as part of # reaching that state, so by the time this tier's first dry-run runs, `StorageProfile` is a known # kind. One edge, no pre-apply. # # WHY NOT just move the CDI CR earlier, into `platform`? Because `platform` is the tier the CNI gates # and everything else waits on: putting CDI's health on that critical path is the mistake the # oauth2-proxy/headlamp splits were made to undo. The CR is correctly a platform-config CR; it is the # StorageProfiles that were in the wrong pass. apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: platform-config-cdi namespace: flux-system spec: interval: 10m # 2m (short, deliberately). kustomize-controller caches API discovery, so on a COLD cluster the # very first attempt here can still fail `no matches for kind "StorageProfile"` even though the CRD # now exists — a stale RESTMapper, not a missing CRD (the failure mode `standup.sh crds` warns # about). The controller resets the mapper and the retry succeeds, so the only cost of that stale # cache is one retryInterval. Keep it short. retryInterval: 2m # 2m is ample: this tier applies two tiny cluster-scoped CRs. There is no workload to roll out, no # image to pull and no controller to become Available, so a slow apply here means a real error, not # a slow rollout — failing fast surfaces it instead of burning 5m per attempt. timeout: 2m # TRUE, and this tier is the SOLE declarer of both StorageProfiles (checked repo-wide), so prune is # component-scoped: the stale set can only ever be a StorageProfile this file stopped rendering. # It is NOT the platform/cilium case — nothing here declares a shared Namespace, or any object a # second path also declares. See the transition note at the bottom of this file. prune: true # TRUE, but understand exactly what it buys and what it does NOT. # # CANNOT STALL: `StorageProfile`'s status type (CDI v1beta1 StorageProfileStatus: storageClass, # provisioner, claimPropertySets, cloneStrategy, dataImportCronSourceFormat) carries NO # `conditions` and NO `observedGeneration`, so kstatus's generic path reports it Current the moment # it is applied. `wait: true` therefore adds a Ready condition without adding a way to hang — which # is why it is safe to let `apps-vms` gate on it (contrast the headlamp gating stall). # If a resource that CAN be unhealthy is ever added to ./platform-config-cdi, re-derive this: # `apps-vms` waits on it. # # WHAT IT DOES NOT PROVE (honest gap): CDI reads the profile's **status**, not its spec, when it # provisions a claim (status.claimPropertySets / status.cloneStrategy), and CDI's # storageprofile-controller computes that status from our spec override plus inference. So Ready # here means "the spec override is accepted by the API server", not "CDI has folded it into # status". That reconcile is a watch-triggered round-trip of seconds, and the dependency requeue # below it is ~30s, so a DataVolume cannot realistically beat it — but it is a timing argument, not # a guarantee, and no `wait` setting can turn it into one (there is no condition to wait on). wait: true dependsOn: # platform-config applies the `CDI` CR and (wait:true) waits for it to be healthy — which is what # causes cdi-operator to create `storageprofiles.cdi.kubevirt.io`. Revision-gated, so this is an # ordering guarantee and not a hope. Transitively this also chains platform -> cilium -> # flux-system, so `cluster-vars` at revision N is applied before this renders revision N. - name: platform-config sourceRef: kind: GitRepository name: flux-system path: ./platform-config-cdi # NO SOPS `decryption` stanza: ../platform/kubevirt/cdi-storageprofiles.yaml is two plain CRs with # no Secret and no *.sops.yaml, so neither overlay needs to patch one in (unlike # oauth2-proxy/headlamp/crafty/velero, which each carry an encrypted Secret). Nothing to add to # clusters/dev or clusters/prod for this tier. # # substituteFrom cluster-vars: the manifest currently contains NO `${TOKEN}`, so this stanza # substitutes nothing today. It is declared anyway for two concrete reasons, not for symmetry: # 1. COVERAGE. conformance/tests/l0-static/flux-postbuild-substitution/check.sh only audits paths # whose Kustomization declares substituteFrom. Omitting it would move # cdi-storageprofiles.yaml OUT of the audited set — a blind spot where a future `${TOKEN}` # could ship as a literal string, precisely the class of bug that check exists to stop. # 2. FAIL-CLOSED. kustomize-controller runs --feature-gates=StrictPostBuildSubstitutions=true, so # with this stanza an undefined token FAILS this Kustomization loudly instead of rendering # empty. Without the stanza there is no substitution at all and the literal ships silently. postBuild: substituteFrom: - kind: ConfigMap name: cluster-vars # ── FIRST RECONCILE AFTER MERGE: a TRANSIENT DELETE-THEN-RECREATE. READ BEFORE MERGING. ────────── # Moving cdi-storageprofiles.yaml out of ./platform-config drops both StorageProfiles from THAT # Kustomization's inventory, and platform-config has `prune: true` — so its next apply DELETES # StorageProfile/longhorn and StorageProfile/longhorn-vm, and this Kustomization recreates them on # the following reconcile (revision-gated: strictly after platform-config's apply, so there is no # dual-ownership fight — the objects are pruned before they are re-applied under this tier's # `kustomize.toolkit.fluxcd.io/name=platform-config-cdi` label). # # Why that is acceptable here — and it is NOT the platform/cilium case: # * Neither object is the sole (or any) declarer of a shared object. No Namespace, no ConfigMap, # nothing else in the repo declares a StorageProfile. The stale set is exactly two # cluster-scoped CDI CRs. # * The objects do not actually go missing: CDI's storageprofile-controller creates a # StorageProfile per StorageClass and recreates a deleted one, with an EMPTY `.spec` and # inferred `.status`. So the transient is "our .spec overrides are absent", not "the object is # absent". # * A StorageProfile is read at DataVolume/PVC PROVISIONING and CLONE time only. Existing PVCs, # existing DataVolumes and running VMs do not consult it again, so nothing already provisioned # is disturbed and no data path is touched. # * The only exposure is a DataVolume/clone STARTED inside the window: it would clone by `copy` # instead of `snapshot`, and on `longhorn-vm` be stamped RWO/Filesystem instead of RWX/Block — # permanently non-live-migratable for that disk. Flux cannot cause that, because `apps-vms` now # dependsOn THIS tier (clusters/base/apps-vms.yaml), so no Flux-managed DataVolume can be # created while the overrides are absent. A human running `virtctl`/`kubectl` in that minute # could. # # PRECONDITION THE OPERATOR MUST CHECK, because it is what bounds the window: the recreate cannot # happen until `platform-config` is Ready, and platform-config runs `wait: true` over the whole # config tier (ClusterIssuers, wildcard Certificate, Gateway, Keycloak, CNPG, KubeVirt, CDI…). On a # healthy cluster that is seconds and the window is ~30-60s (kustomize-controller's # --requeue-dependency default is 30s). If platform-config is NOT Ready when this merges, the # StorageProfiles are pruned and stay at CDI defaults until it recovers. So: # `flux get kustomization platform-config` must be Ready BEFORE merging. # If it is not, do the two-step the cilium move used instead: merge with `platform-config.prune` # temporarily set to `false` (the objects are then dropped from its inventory but NOT deleted, and # this tier ADOPTS them with zero window), verify the ownership label transferred, then restore # `prune: true` in a second commit.