diff --git a/clusters/README.md b/clusters/README.md index 633320c..7ed569e 100644 --- a/clusters/README.md +++ b/clusters/README.md @@ -97,6 +97,27 @@ Kustomization: apps-wave-1 (./apps/wave-1, dependsOn apps-wave-0) Kustomization: apps-wave-N (./apps/wave-N, dependsOn apps-wave-N-1) ``` +### Second-order CRDs need a THIRD pass (`platform-config-cdi`, added 2026-09-05) + +The controllers/configs split (`platform` → `platform-config`) assumes a CRD is registered by the +controller **install**. CDI breaks that: `cdi-operator.yaml` registers only `cdis.cdi.kubevirt.io`, +and cdi-operator creates `storageprofiles` / `datavolumes` / `datasources` **only after it has +reconciled the `CDI` CR**. Since kustomize-controller server-side **dry-runs every object in a pass +before applying any of them**, having the `CDI` CR and the CDI `StorageProfile`s in one pass was a +**deadlock, not a race**: unknown kind → pass aborts → CR never applied → CRD never created → +`platform-config` can never converge → every tier below it blocked. Invisible on dev (its CDI CRDs +have existed for months); it cost 12h on prod's cold start. + +So there is a third tier, `platform-config-cdi` (`./platform-config-cdi`, `dependsOn: +platform-config`), and a rule: + +> A CR belongs in `platform-config-cdi`, not `platform-config`, when the CRD it needs is created by a +> **controller at runtime** rather than by a manifest in git. + +`apps-vms` `dependsOn` it as well as `platform-config`, because CDI reads +`StorageProfile/longhorn-vm` at DataVolume **provisioning** time — a VM disk provisioned before that +profile exists is stamped RWO/Filesystem and is permanently not live-migratable. + Wave 0 `dependsOn` `platform-config` (transitively `platform`) so CNI/storage/TLS/DNS/gateway exist before any workload reconciles. The app tier is split into dependency-ordered WAVES (`catalog/render.py --emit-waves`): each `apps-wave-` health-gates (wait:true) ONLY its own diff --git a/clusters/base/apps-vms.yaml b/clusters/base/apps-vms.yaml index 40e032a..32bab17 100644 --- a/clusters/base/apps-vms.yaml +++ b/clusters/base/apps-vms.yaml @@ -4,8 +4,11 @@ # NON-GATING (wait:false): unlike the app waves, this does NOT health-wait. A VM's first-boot image # import (CDI, minutes) + in-guest software install mean a VM can be intentionally long-not-Ready, so # health-gating it would wedge reconciliation for no benefit. VMs have no in-cluster dependents. -# dependsOn platform-config so KubeVirt/CDI + the longhorn-vm StorageClass + CDI StorageProfiles -# exist first. prune:true — this Kustomization solely owns the VM manifests (no adoption concern). +# dependsOn platform-config so KubeVirt/CDI + the longhorn-vm StorageClass exist first, AND +# platform-config-cdi so the CDI StorageProfiles do (2026-09-05: the StorageProfiles MOVED OUT of +# platform-config into their own tier, so `dependsOn: platform-config` alone NO LONGER guarantees +# them — see the dependsOn block). prune:true — this Kustomization solely owns the VM manifests (no +# adoption concern). apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: @@ -19,6 +22,28 @@ spec: wait: false dependsOn: - name: platform-config + # LOAD-BEARING, not decoration. apps/vms/unifi/20-datavolume.yaml provisions a DataVolume on + # storageClassName `longhorn-vm`, and CDI reads StorageProfile/longhorn-vm AT PROVISIONING TIME to + # decide the claim's accessModes/volumeMode. Without that profile's `claimPropertySets` + # (ReadWriteMany + Block) the PVC is stamped RWO/Filesystem and the VM disk is PERMANENTLY not + # live-migratable — a silent, unrecoverable-without-recreate loss. The profiles used to arrive + # with platform-config; since 2026-09-05 they arrive with `platform-config-cdi`, so the edge has + # to move with them. platform-config-cdi runs wait:true and StorageProfile has no status + # conditions (kstatus -> Current on apply), so this edge cannot stall this tier. + # + # HONEST LIMIT: CDI provisions from the profile's STATUS, which its storageprofile-controller + # derives from our spec. This edge proves the spec is applied, not that CDI has folded it into + # status — that is a watch-triggered round-trip of seconds against a ~30s dependency requeue, so + # it is a timing argument rather than a guarantee. See clusters/base/platform-config-cdi.yaml. + # + # This edge also makes the StorageProfile prune/recreate transition safe: platform-config's prune + # deletes the two profiles for the ~30-60s before platform-config-cdi re-applies them, and this + # dependsOn is what guarantees Flux cannot create a DataVolume inside that window. + # + # The platform-config edge above is kept although platform-config-cdi already dependsOn it: the + # edges are a set, there is no cycle, and stating both means a future edit to platform-config-cdi + # cannot silently drop this tier's KubeVirt/StorageClass gate. + - name: platform-config-cdi sourceRef: kind: GitRepository name: flux-system diff --git a/clusters/base/kustomization.yaml b/clusters/base/kustomization.yaml index b1db32b..6f56415 100644 --- a/clusters/base/kustomization.yaml +++ b/clusters/base/kustomization.yaml @@ -14,6 +14,15 @@ resources: - cilium.yaml # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true) - platform.yaml # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs) - platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform) + # SECOND-ORDER CDI configs — CRs whose CRD is created by a controller AT RUNTIME, not by any + # manifest in git. cdi-operator.yaml registers only `cdis.cdi.kubevirt.io`; `storageprofiles` (and + # datavolumes, datasources, …) appear only after cdi-operator reconciles the `CDI` CR that + # platform-config applies. Because Flux dry-runs every object in a pass before applying any of them, + # keeping the StorageProfiles in platform-config was a TRUE DEADLOCK (unknown kind -> pass aborts -> + # CR never applied -> CRD never created -> tier never converges), invisible on dev and fatal on any + # cold cluster — observed on prod 2026-09-05. This tier is the durable fix; `standup.sh crds` stays + # as the belt-and-braces repair for clusters already wedged. See clusters/base/platform-config-cdi.yaml. + - platform-config-cdi.yaml # Flux Kustomization -> ./platform-config-cdi (dependsOn platform-config) # App tier, split into dependency-ordered WAVES (was the single ./apps `apps` Kustomization). # Each apps-wave- dependsOn wave N-1 (wave 0 dependsOn platform-config) and health-gates # (wait:true) ONLY its own wave, so a crashloop in a later wave can't stall an earlier one. diff --git a/clusters/base/platform-config-cdi.yaml b/clusters/base/platform-config-cdi.yaml new file mode 100644 index 0000000..6bf985b --- /dev/null +++ b/clusters/base/platform-config-cdi.yaml @@ -0,0 +1,135 @@ +--- +# Flux Kustomization: SECOND-ORDER CDI CONFIGS — reconciles ./platform-config-cdi. +# Serves R-GITOPS-01 (all desired state in git, reconciled by Flux) and R-ENV-02. +# +# WHY THIS TIER EXISTS — the durable fix for the 2026-09-05 prod CDI DEADLOCK +# +# `platform-config` applied the `CDI` CR and the CDI `StorageProfile`s in the SAME pass. +# kustomize-controller server-side DRY-RUNS EVERY object before applying ANY of them, and +# `storageprofiles.cdi.kubevirt.io` does not exist until cdi-operator has reconciled that `CDI` CR. +# So the unknown kind aborted the whole tier → the CR was never applied → the CRD was never created → +# platform-config could NEVER converge and every tier below it stayed blocked. Not a race: a +# deadlock. cdi-operator was Available for 12h on prod with no CR. Invisible on dev (its CDI CRDs +# have existed for months), FATAL on any cold cluster. See DECISIONS-LOG 2026-09-05. +# +# `bootstrap/standup.sh crds` pre-applies the `CDI` CR to break it by hand. That is a WORKAROUND and +# stays as belt-and-braces for clusters already wedged; THIS Kustomization is the fix that makes the +# graph order itself, so a cold cluster needs no human in the loop. +# +# THE ORDERING GUARANTEE, precisely: `dependsOn` is REVISION-gated, not merely Ready-gated (verified +# against kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies`, our +# version): sharing a GitRepository with the dependency, this Kustomization cannot reconcile revision +# N until `platform-config` has APPLIED revision N — and, because platform-config runs `wait: true`, +# not until the `CDI` CR it applied is HEALTHY. cdi-operator creates the remaining CDI CRDs as part of +# reaching that state, so by the time this tier's first dry-run runs, `StorageProfile` is a known +# kind. One edge, no pre-apply. +# +# WHY NOT just move the CDI CR earlier, into `platform`? Because `platform` is the tier the CNI gates +# and everything else waits on: putting CDI's health on that critical path is the mistake the +# oauth2-proxy/headlamp splits were made to undo. The CR is correctly a platform-config CR; it is the +# StorageProfiles that were in the wrong pass. +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: platform-config-cdi + namespace: flux-system +spec: + interval: 10m + # 2m (short, deliberately). kustomize-controller caches API discovery, so on a COLD cluster the + # very first attempt here can still fail `no matches for kind "StorageProfile"` even though the CRD + # now exists — a stale RESTMapper, not a missing CRD (the failure mode `standup.sh crds` warns + # about). The controller resets the mapper and the retry succeeds, so the only cost of that stale + # cache is one retryInterval. Keep it short. + retryInterval: 2m + # 2m is ample: this tier applies two tiny cluster-scoped CRs. There is no workload to roll out, no + # image to pull and no controller to become Available, so a slow apply here means a real error, not + # a slow rollout — failing fast surfaces it instead of burning 5m per attempt. + timeout: 2m + # TRUE, and this tier is the SOLE declarer of both StorageProfiles (checked repo-wide), so prune is + # component-scoped: the stale set can only ever be a StorageProfile this file stopped rendering. + # It is NOT the platform/cilium case — nothing here declares a shared Namespace, or any object a + # second path also declares. See the transition note at the bottom of this file. + prune: true + # TRUE, but understand exactly what it buys and what it does NOT. + # + # CANNOT STALL: `StorageProfile`'s status type (CDI v1beta1 StorageProfileStatus: storageClass, + # provisioner, claimPropertySets, cloneStrategy, dataImportCronSourceFormat) carries NO + # `conditions` and NO `observedGeneration`, so kstatus's generic path reports it Current the moment + # it is applied. `wait: true` therefore adds a Ready condition without adding a way to hang — which + # is why it is safe to let `apps-vms` gate on it (contrast the headlamp gating stall). + # If a resource that CAN be unhealthy is ever added to ./platform-config-cdi, re-derive this: + # `apps-vms` waits on it. + # + # WHAT IT DOES NOT PROVE (honest gap): CDI reads the profile's **status**, not its spec, when it + # provisions a claim (status.claimPropertySets / status.cloneStrategy), and CDI's + # storageprofile-controller computes that status from our spec override plus inference. So Ready + # here means "the spec override is accepted by the API server", not "CDI has folded it into + # status". That reconcile is a watch-triggered round-trip of seconds, and the dependency requeue + # below it is ~30s, so a DataVolume cannot realistically beat it — but it is a timing argument, not + # a guarantee, and no `wait` setting can turn it into one (there is no condition to wait on). + wait: true + dependsOn: + # platform-config applies the `CDI` CR and (wait:true) waits for it to be healthy — which is what + # causes cdi-operator to create `storageprofiles.cdi.kubevirt.io`. Revision-gated, so this is an + # ordering guarantee and not a hope. Transitively this also chains platform -> cilium -> + # flux-system, so `cluster-vars` at revision N is applied before this renders revision N. + - name: platform-config + sourceRef: + kind: GitRepository + name: flux-system + path: ./platform-config-cdi + # NO SOPS `decryption` stanza: ../platform/kubevirt/cdi-storageprofiles.yaml is two plain CRs with + # no Secret and no *.sops.yaml, so neither overlay needs to patch one in (unlike + # oauth2-proxy/headlamp/crafty/velero, which each carry an encrypted Secret). Nothing to add to + # clusters/dev or clusters/prod for this tier. + # + # substituteFrom cluster-vars: the manifest currently contains NO `${TOKEN}`, so this stanza + # substitutes nothing today. It is declared anyway for two concrete reasons, not for symmetry: + # 1. COVERAGE. conformance/tests/l0-static/flux-postbuild-substitution/check.sh only audits paths + # whose Kustomization declares substituteFrom. Omitting it would move + # cdi-storageprofiles.yaml OUT of the audited set — a blind spot where a future `${TOKEN}` + # could ship as a literal string, precisely the class of bug that check exists to stop. + # 2. FAIL-CLOSED. kustomize-controller runs --feature-gates=StrictPostBuildSubstitutions=true, so + # with this stanza an undefined token FAILS this Kustomization loudly instead of rendering + # empty. Without the stanza there is no substitution at all and the literal ships silently. + postBuild: + substituteFrom: + - kind: ConfigMap + name: cluster-vars + # ── FIRST RECONCILE AFTER MERGE: a TRANSIENT DELETE-THEN-RECREATE. READ BEFORE MERGING. ────────── + # Moving cdi-storageprofiles.yaml out of ./platform-config drops both StorageProfiles from THAT + # Kustomization's inventory, and platform-config has `prune: true` — so its next apply DELETES + # StorageProfile/longhorn and StorageProfile/longhorn-vm, and this Kustomization recreates them on + # the following reconcile (revision-gated: strictly after platform-config's apply, so there is no + # dual-ownership fight — the objects are pruned before they are re-applied under this tier's + # `kustomize.toolkit.fluxcd.io/name=platform-config-cdi` label). + # + # Why that is acceptable here — and it is NOT the platform/cilium case: + # * Neither object is the sole (or any) declarer of a shared object. No Namespace, no ConfigMap, + # nothing else in the repo declares a StorageProfile. The stale set is exactly two + # cluster-scoped CDI CRs. + # * The objects do not actually go missing: CDI's storageprofile-controller creates a + # StorageProfile per StorageClass and recreates a deleted one, with an EMPTY `.spec` and + # inferred `.status`. So the transient is "our .spec overrides are absent", not "the object is + # absent". + # * A StorageProfile is read at DataVolume/PVC PROVISIONING and CLONE time only. Existing PVCs, + # existing DataVolumes and running VMs do not consult it again, so nothing already provisioned + # is disturbed and no data path is touched. + # * The only exposure is a DataVolume/clone STARTED inside the window: it would clone by `copy` + # instead of `snapshot`, and on `longhorn-vm` be stamped RWO/Filesystem instead of RWX/Block — + # permanently non-live-migratable for that disk. Flux cannot cause that, because `apps-vms` now + # dependsOn THIS tier (clusters/base/apps-vms.yaml), so no Flux-managed DataVolume can be + # created while the overrides are absent. A human running `virtctl`/`kubectl` in that minute + # could. + # + # PRECONDITION THE OPERATOR MUST CHECK, because it is what bounds the window: the recreate cannot + # happen until `platform-config` is Ready, and platform-config runs `wait: true` over the whole + # config tier (ClusterIssuers, wildcard Certificate, Gateway, Keycloak, CNPG, KubeVirt, CDI…). On a + # healthy cluster that is seconds and the window is ~30-60s (kustomize-controller's + # --requeue-dependency default is 30s). If platform-config is NOT Ready when this merges, the + # StorageProfiles are pruned and stay at CDI defaults until it recovers. So: + # `flux get kustomization platform-config` must be Ready BEFORE merging. + # If it is not, do the two-step the cilium move used instead: merge with `platform-config.prune` + # temporarily set to `false` (the objects are then dropped from its inventory but NOT deleted, and + # this tier ADOPTS them with zero window), verify the ownership label transferred, then restore + # `prune: true` in a second commit. diff --git a/infra/PROD-STANDUP.md b/infra/PROD-STANDUP.md index d9a342c..c896007 100644 --- a/infra/PROD-STANDUP.md +++ b/infra/PROD-STANDUP.md @@ -141,10 +141,52 @@ before applying any of them, so one unknown `kind` aborts the entire tier.** Flux. It carries no `${}` tokens, so applying it here cannot diverge from what Flux would render. The CRD appeared in 10s. - **The durable fix is a repo change, not this script:** move `cdi-storageprofiles.yaml` out of - `platform-config` into its own Kustomization with `dependsOn: platform-config`, so the CR lands one - reconcile before the resources that need its CRDs. Until that lands, C2 is a prerequisite for C5 on - any cold cluster. + **REPAIRED IN THE GRAPH (2026-09-05).** `cdi-storageprofiles.yaml` now lives in its own tier, + `./platform-config-cdi`, with `dependsOn: platform-config` and `wait: true`. Because `dependsOn` is + **revision**-gated and `platform-config` runs `wait: true`, the CDI CR must be applied *and healthy* + — which is precisely the event that creates the `StorageProfile` CRD — before the profiles are + applied. A cold cluster now orders itself. The `crds` step is retained as a fallback for a cluster + already stuck in the broken state, and because it is idempotent. + + **The generalisable rule, now recorded in `clusters/README.md`:** a CR whose CRD is created by a + *controller at runtime* is **second-order** and needs its own pass. `cdi-operator.yaml` registers + exactly one CRD (`cdis.cdi.kubevirt.io`); `storageprofiles`, `datavolumes`, `datasources` and + `dataimportcrons` appear only after cdi-operator reconciles the CDI CR. An audit of + `platform-config` found `StorageProfile` is the only second-order kind there. + + `apps-vms` gained `dependsOn: platform-config-cdi` — load-bearing, not cosmetic: + `apps/vms/unifi/20-datavolume.yaml` provisions on `storageClassName: longhorn-vm`, and its comment + previously claimed `platform-config` guaranteed the profiles, which the move would have silently + falsified. + +#### Merge precondition and the one-time transient + +`platform-config` has `prune: true`, so on the first reconcile after merge it **deletes** both +StorageProfiles, CDI recreates them within seconds with an empty spec, and `platform-config-cdi` +re-applies the overrides ~30–60s later (kustomize-controller's dependency requeue). + +**Measured on dev, so the exposure is known rather than assumed** — comparing our override against +what CDI infers unaided for the same Longhorn provisioner: + +| profile | `spec` (ours) | CDI-derived `status` | +|---|---|---| +| `longhorn-vm` (we override) | RWX / Block, `snapshot` | **ReadWriteMany** / Block, `snapshot` | +| `longhorn-static` (no override) | *unset* | **ReadWriteOnce** / Block, `snapshot` | + +So **`cloneStrategy` is not at risk** — CDI infers `snapshot` for Longhorn anyway. **`accessModes` +is:** unaided, CDI produces **RWO**. A DataVolume provisioned on `longhorn-vm` *inside the window* +would be stamped RWO/Block and be **permanently non-live-migratable**. Flux can no longer cause that +(`apps-vms` now gates on the new tier); **do not start a VM disk import, clone or DataVolume by hand +during the window.** + +- [ ] **Before merging, confirm `flux get kustomization platform-config` is Ready on the target.** If + it is not, the profiles are pruned and sit at CDI defaults until it recovers — and prod was in + exactly that state hours earlier. The fallback is the two-step the Cilium move used: merge with + `platform-config.prune: false`, confirm the ownership label transferred, restore `prune: true` in + a second commit. +- [ ] After: both profiles carry `spec.cloneStrategy: snapshot`, `longhorn-vm` carries + `claimPropertySets` RWX/Block, and both are labelled + `kustomize.toolkit.fluxcd.io/name=platform-config-cdi`. ### C3 — unwedge stuck Helm releases diff --git a/platform-config-cdi/kustomization.yaml b/platform-config-cdi/kustomization.yaml new file mode 100644 index 0000000..035b91f --- /dev/null +++ b/platform-config-cdi/kustomization.yaml @@ -0,0 +1,41 @@ +--- +# platform-config-cdi — the SECOND-ORDER config wave: custom resources whose CRD is created at +# RUNTIME by a controller that a `platform-config` custom resource switches on. +# +# ── THE BUG THIS DIRECTORY EXISTS TO FIX (observed on PROD, 2026-09-05) ─────────────────────────── +# The repo already splits CONTROLLERS (`platform`) from CONFIGS (`platform-config`) because a CR +# cannot be dry-run before its CRD is registered. That split assumes a CRD is registered by the +# controller INSTALL. CDI breaks the assumption: `cdi-operator.yaml` registers exactly ONE CRD +# (`cdis.cdi.kubevirt.io`). Every OTHER CDI CRD — `storageprofiles`, `datavolumes`, `datasources`, +# `dataimportcrons`, … — is created by cdi-operator only AFTER it has reconciled the `CDI` CR. +# +# So `cdi-cr.yaml` is a FIRST-order CR (its CRD comes from the install) and belongs in +# platform-config, while `cdi-storageprofiles.yaml` is a SECOND-order CR (its CRD comes from +# reconciling the first-order CR). Having both in ONE Flux Kustomization is a true DEADLOCK, not a +# race: kustomize-controller server-side dry-runs EVERY object before applying ANY of them, so the +# unknown `StorageProfile` kind aborts the whole pass → the `CDI` CR is never applied → the +# StorageProfile CRD is never created → platform-config can NEVER converge, and every tier that +# dependsOn it stays blocked forever. On prod, cdi-operator sat Available for 12h with no CR: +# +# StorageProfile/longhorn dry-run failed: no matches for kind "StorageProfile" +# in version "cdi.kubevirt.io/v1beta1" +# +# Invisible on dev (whose CDI CRDs have existed for months) and fatal on any COLD cluster. +# +# ── THE RULE, so this is not rediscovered a third time ──────────────────────────────────────────── +# A CR belongs HERE (not in platform-config) when the CRD it needs is created by a CONTROLLER AT +# RUNTIME rather than by a manifest in git. Grep test: if `grep -rn "kind: CustomResourceDefinition"` +# over ./platform (following remote bases) does not produce the CRD, and no chart installs it, then +# some operator creates it after being switched on — and this is the tier for it. +# +# Files are referenced IN PLACE via ../platform/... (Flux builds with LoadRestrictionsNone, and the +# manifests stay beside the component they configure). Build/verify locally with: +# kubectl kustomize --load-restrictor LoadRestrictionsNone platform-config-cdi +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + # CDI StorageProfiles — snapshot smart-clone for `longhorn`, and RWX/Block + snapshot clone for the + # `longhorn-vm` virt-default class (clears CDIDefaultStorageClassDegraded). `StorageProfile`'s CRD + # is created by cdi-operator only once the `CDI` CR (applied by platform-config) is reconciled, + # which is exactly why these cannot live in the same pass as that CR. + - ../platform/kubevirt/cdi-storageprofiles.yaml diff --git a/platform-config/kustomization.yaml b/platform-config/kustomization.yaml index 6f7e9e4..cc4a893 100644 --- a/platform-config/kustomization.yaml +++ b/platform-config/kustomization.yaml @@ -41,9 +41,23 @@ resources: # ConfigMap kept beside them for locality. - ../platform/kubevirt/kubevirt-cr.yaml - ../platform/kubevirt/cdi-cr.yaml - # CDI StorageProfiles: snapshot smart-clone for `longhorn`, and RWX/Block + snapshot clone for the - # `longhorn-vm` virt-default class (consume cdi.kubevirt.io; clears CDIDefaultStorageClassDegraded). - - ../platform/kubevirt/cdi-storageprofiles.yaml + # *** DO NOT ADD ../platform/kubevirt/cdi-storageprofiles.yaml BACK HERE. IT DEADLOCKS THIS TIER. *** + # The CDI StorageProfiles were MOVED OUT on 2026-09-05 into their own Flux Kustomization + # (clusters/base/platform-config-cdi.yaml -> ./platform-config-cdi, dependsOn platform-config). + # + # `StorageProfile`'s CRD is NOT registered by the CDI operator INSTALL — cdi-operator.yaml ships only + # `cdis.cdi.kubevirt.io`. cdi-operator creates `storageprofiles` (and datavolumes, datasources, …) + # only AFTER it has reconciled the `CDI` CR two lines above. Flux server-side DRY-RUNS EVERY object + # in a Kustomization before applying ANY of them, so having both in this list meant: unknown kind + # `StorageProfile` -> the whole pass aborts -> the `CDI` CR is never applied -> the CRD is never + # created -> THIS TIER CAN NEVER CONVERGE, and every tier that dependsOn it is blocked forever. + # A true deadlock, not a race. Observed on prod 2026-09-05 (cdi-operator Available 12h, no CR): + # StorageProfile/longhorn dry-run failed: no matches for kind "StorageProfile" in version + # "cdi.kubevirt.io/v1beta1" + # Invisible on dev, whose CDI CRDs have existed for months. See DECISIONS-LOG 2026-09-05. + # + # THE RULE: a CR whose CRD is created by a controller AT RUNTIME (rather than by a manifest in git) + # is SECOND-ORDER and belongs in ./platform-config-cdi, not here. - ../platform/kubevirt/servicemonitor.yaml - ../platform/kubevirt/alerts.yaml - ../platform/kubevirt/dashboard.yaml diff --git a/platform/external-snapshotter/README.md b/platform/external-snapshotter/README.md index 1819543..c9378f4 100644 --- a/platform/external-snapshotter/README.md +++ b/platform/external-snapshotter/README.md @@ -20,8 +20,12 @@ Neither k3s nor the Longhorn chart ships this. It is a prerequisite for: ## Wiring - **Pass:** `platform` (controllers). Registers the CRDs. -- The `VolumeSnapshotClass` / `StorageProfile` CRs that consume the CRDs live in the - `platform-config` pass (`dependsOn: platform`), per the controllers/configs split. +- The `VolumeSnapshotClass` CR that consumes the CRDs lives in the `platform-config` pass + (`dependsOn: platform`), per the controllers/configs split. +- The CDI `StorageProfile`s live one pass further out, in `platform-config-cdi` + (`dependsOn: platform-config`) — `storageprofiles.cdi.kubevirt.io` is created by cdi-operator only + after it reconciles the `CDI` CR, so they cannot share a pass with it. See + `clusters/base/platform-config-cdi.yaml`. ## Version diff --git a/platform/external-snapshotter/kustomization.yaml b/platform/external-snapshotter/kustomization.yaml index 9c885ad..8bfd196 100644 --- a/platform/external-snapshotter/kustomization.yaml +++ b/platform/external-snapshotter/kustomization.yaml @@ -9,8 +9,10 @@ # * no VolumeSnapshotClass can be created (item 2), and # * CDI cannot do SNAPSHOT smart-clone (item 3) — it falls back to slow host-assisted `copy`, # which leaves the default virt StorageClass "degraded" (CDIDefaultStorageClassDegraded). -# Installing it here (controllers pass) registers the CRDs; the VolumeSnapshotClass + the CDI -# StorageProfiles that CONSUME them are applied by platform-config (dependsOn: platform). +# Installing it here (controllers pass) registers the CRDs; the VolumeSnapshotClass that CONSUMES them +# is applied by platform-config (dependsOn: platform), and the CDI StorageProfiles that set +# `cloneStrategy: snapshot` by platform-config-cdi (dependsOn: platform-config — their own CRD is +# created by cdi-operator only after the `CDI` CR reconciles; see clusters/base/platform-config-cdi.yaml). # # CONTROLLERS/CONFIGS SPLIT (F-MON1): this is the CONTROLLERS half. The snapshot.storage.k8s.io # CRDs are registered by the `client/config/crd` base below; the snapshot-controller Deployment diff --git a/platform/kubevirt/cdi-storageprofiles.yaml b/platform/kubevirt/cdi-storageprofiles.yaml index f9eed35..b65b6ff 100644 --- a/platform/kubevirt/cdi-storageprofiles.yaml +++ b/platform/kubevirt/cdi-storageprofiles.yaml @@ -1,9 +1,19 @@ --- # CDI StorageProfiles — the storage "recipes" CDI uses when it provisions/clones VM disk PVCs -# (ADR-0005). CONSUME the cdi.kubevirt.io CRD registered by the CDI operator (platform pass), so — -# like cdi-cr.yaml — these CRs are applied by the `platform-config` Flux Kustomization -# (dependsOn: platform). They live here beside the KubeVirt/CDI component and are referenced in -# place from platform-config/kustomization.yaml. +# (ADR-0005). +# +# APPLIED BY `platform-config-cdi`, NOT `platform-config` (moved 2026-09-05). These are SECOND-ORDER +# CRs: `storageprofiles.cdi.kubevirt.io` is NOT registered by the CDI operator install (cdi-operator.yaml +# ships only `cdis.cdi.kubevirt.io`) — cdi-operator creates it only AFTER it has reconciled the `CDI` CR +# that platform-config applies. Flux server-side dry-runs every object in a pass before applying any of +# them, so co-locating these with that CR was a TRUE DEADLOCK: unknown kind -> pass aborts -> CR never +# applied -> CRD never created -> platform-config never converges -> every tier below it blocked. It was +# invisible on dev (CDI CRDs long present) and cost 12h on prod's cold start, 2026-09-05. +# They live here beside the KubeVirt/CDI component and are referenced in place from +# platform-config-cdi/kustomization.yaml (dependsOn: platform-config, revision-gated). +# +# `apps-vms` dependsOn platform-config-cdi for these: a DataVolume created before the longhorn-vm +# profile below exists is stamped RWO/Filesystem and is permanently not live-migratable. # # CDI AUTO-CREATES a StorageProfile per StorageClass and fills its STATUS (inferred capabilities). # The operator owns `.status`; the USER owns `.spec`. Setting `.spec` here does NOT fight CDI — it diff --git a/platform/kubevirt/kustomization.yaml b/platform/kubevirt/kustomization.yaml index d546d91..9d1afa5 100644 --- a/platform/kubevirt/kustomization.yaml +++ b/platform/kubevirt/kustomization.yaml @@ -8,6 +8,16 @@ # (dependsOn: platform, wait) so the CRDs exist BEFORE the CRs apply. Applying both in one atomic pass # deadlocks ("no matches for kind KubeVirt"), exactly the failure F-MON1 documents. # +# THERE IS A THIRD PASS FOR CDI, and it is not optional (2026-09-05). The two operator manifests below +# register exactly ONE CDI CRD, `cdis.cdi.kubevirt.io`. Every other CDI CRD — `storageprofiles`, +# `datavolumes`, `datasources`, `dataimportcrons` — is created by cdi-operator only AFTER it reconciles +# the `CDI` CR. So `cdi-storageprofiles.yaml` is a SECOND-ORDER CR and CANNOT share a pass with +# `cdi-cr.yaml`: the dry-run of the unknown `StorageProfile` kind aborts the pass, the CR is never +# applied, the CRD is never created, and platform-config deadlocks (cost 12h on prod's cold start). +# It is therefore applied by a third Kustomization, `platform-config-cdi` (dependsOn platform-config). +# Before adding any new CDI CR here, ask which pass owns it: CRD in git -> platform-config; +# CRD created by cdi-operator at runtime -> platform-config-cdi. +# # Namespaces: the operator manifests each create their own namespace (`kubevirt`, `cdi`) with the # pod-security profile they need (kubevirt is privileged) — so we do NOT redeclare them here (that # would be a duplicate-resource kustomize error). These are PLATFORM namespaces: they intentionally