fix(gitops): CDI cold-start deadlock repaired IN THE GRAPH, not by a pre-apply
`platform-config` applied the CDI CR *and* cdi-storageprofiles.yaml in ONE pass.
StorageProfile's CRD does not exist until cdi-operator reconciles that CR, and
Flux server-side dry-runs every object before applying any — so the unknown kind
aborted the tier, the CR was never applied, the CRD was never created, and the
tier could NEVER converge. Observed on prod: cdi-operator Available 12h, no CDI
CR, no storageprofiles CRD, every downstream tier blocked.
The profiles now live in their own tier ./platform-config-cdi with
dependsOn: platform-config and wait: true. Because dependsOn is REVISION-gated
and platform-config runs wait: true, the CDI CR must be applied AND healthy —
exactly the event that creates the CRD — before the profiles are applied. Cold
clusters order themselves. `standup.sh crds` is retained as a fallback for
clusters already stuck, and because it is idempotent.
THE GENERALISABLE RULE (recorded in clusters/README.md): a CR whose CRD is
created by a CONTROLLER AT RUNTIME is second-order and needs its own pass.
cdi-operator.yaml registers exactly ONE CRD (cdis.cdi.kubevirt.io); storageprofiles,
datavolumes, datasources and dataimportcrons appear only after it reconciles the
CDI CR. An audit found StorageProfile is the only second-order kind in
platform-config — KubeVirt, CDI, KeycloakRealmImport, CNPG Cluster,
VolumeSnapshotClass, ClusterIssuer, PrometheusRule, ServiceMonitor and the Cilium
L2 CRs all have their CRD in git or in a chart.
Moving the CDI CR up into `platform` was rejected: that would put CDI's health on
the tier the CNI gates and everything waits on — the mistake the oauth2-proxy and
headlamp splits exist to undo.
apps-vms gains dependsOn: platform-config-cdi. LOAD-BEARING, not cosmetic:
apps/vms/unifi/20-datavolume.yaml provisions on storageClassName: longhorn-vm and
its comment claimed platform-config guaranteed the profiles — which this move
would have silently falsified.
THE PRUNE TRANSIENT, MEASURED RATHER THAN ASSUMED. platform-config has
prune: true, so it deletes both profiles once, CDI recreates them with empty spec,
and the new tier re-applies the overrides ~30-60s later. Compared our override
against CDI's unaided inference for the same Longhorn provisioner on dev:
longhorn-vm (we override) spec RWX/Block -> status ReadWriteMany/Block
longhorn-static (no override) spec unset -> status ReadWriteOnce/Block
Both infer cloneStrategy: snapshot. So cloneStrategy is NOT at risk; accessModes
IS — unaided CDI gives RWO. A DataVolume provisioned on longhorn-vm inside the
window would be stamped RWO/Block and be PERMANENTLY non-live-migratable. Flux can
no longer cause it (apps-vms gates on the new tier); a human must not do it by hand.
Corroborating evidence that the recreate happens at all: dev has 5 StorageClasses
and 5 StorageProfiles, three with fluxOwner=NONE, spec unset and status populated
— CDI creating and inferring them with no Flux involvement.
MERGE PRECONDITION: `flux get kustomization platform-config` must be Ready on the
target first. If it is not, the profiles are pruned and sit at CDI defaults until
it recovers — and prod was in that state hours earlier. Fallback is the two-step
the Cilium move used (merge with prune: false, verify the ownership label moved,
restore prune: true).
substituteFrom: cluster-vars is declared even though the manifest has no ${TOKEN},
deliberately: conformance/tests/l0-static/flux-postbuild-substitution only audits
paths whose Kustomization declares it, so omitting it would move the file OUT of
the audited set — a blind spot for a future literal-token ship. Gate confirms it
stayed in (path=./platform-config-cdi files=2 defined=0 hazards=0). No SOPS
decryption stanza: the moved file is two plain CRs, so no overlay change was needed.
VERIFIED INDEPENDENTLY OF THE AGENT: platform-config renders 0 StorageProfile
while keeping the CDI and KubeVirt CRs; platform-config-cdi renders exactly the
two profiles; the graph is byte-identical across dev and prod with apps-vms
carrying both edges and no cycle. Full CI gate PASS.
KNOWN FOLLOW-UP: bootstrap/standup.sh TIER_ORDER does not yet include
platform-config-cdi, so `resume` will not wait on it. Another agent holds that
file; the PM will add it.
Refs R-GITOPS-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
c8c1d2935d
commit
834928c237
10 files changed
+284
-13
No files matched your search
@@ -97,6 +97,27 @@ Kustomization: apps-wave-1 (./apps/wave-1, dependsOn apps-wave-0)
|
||||
Kustomization: apps-wave-N (./apps/wave-N, dependsOn apps-wave-N-1)
|
||||
```
|
||||
|
||||
### Second-order CRDs need a THIRD pass (`platform-config-cdi`, added 2026-09-05)
|
||||
|
||||
The controllers/configs split (`platform` → `platform-config`) assumes a CRD is registered by the
|
||||
controller **install**. CDI breaks that: `cdi-operator.yaml` registers only `cdis.cdi.kubevirt.io`,
|
||||
and cdi-operator creates `storageprofiles` / `datavolumes` / `datasources` **only after it has
|
||||
reconciled the `CDI` CR**. Since kustomize-controller server-side **dry-runs every object in a pass
|
||||
before applying any of them**, having the `CDI` CR and the CDI `StorageProfile`s in one pass was a
|
||||
**deadlock, not a race**: unknown kind → pass aborts → CR never applied → CRD never created →
|
||||
`platform-config` can never converge → every tier below it blocked. Invisible on dev (its CDI CRDs
|
||||
have existed for months); it cost 12h on prod's cold start.
|
||||
|
||||
So there is a third tier, `platform-config-cdi` (`./platform-config-cdi`, `dependsOn:
|
||||
platform-config`), and a rule:
|
||||
|
||||
> A CR belongs in `platform-config-cdi`, not `platform-config`, when the CRD it needs is created by a
|
||||
> **controller at runtime** rather than by a manifest in git.
|
||||
|
||||
`apps-vms` `dependsOn` it as well as `platform-config`, because CDI reads
|
||||
`StorageProfile/longhorn-vm` at DataVolume **provisioning** time — a VM disk provisioned before that
|
||||
profile exists is stamped RWO/Filesystem and is permanently not live-migratable.
|
||||
|
||||
Wave 0 `dependsOn` `platform-config` (transitively `platform`) so CNI/storage/TLS/DNS/gateway
|
||||
exist before any workload reconciles. The app tier is split into dependency-ordered WAVES
|
||||
(`catalog/render.py --emit-waves`): each `apps-wave-<N>` health-gates (wait:true) ONLY its own
|
||||
|
||||
@@ -4,8 +4,11 @@
|
||||
# NON-GATING (wait:false): unlike the app waves, this does NOT health-wait. A VM's first-boot image
|
||||
# import (CDI, minutes) + in-guest software install mean a VM can be intentionally long-not-Ready, so
|
||||
# health-gating it would wedge reconciliation for no benefit. VMs have no in-cluster dependents.
|
||||
# dependsOn platform-config so KubeVirt/CDI + the longhorn-vm StorageClass + CDI StorageProfiles
|
||||
# exist first. prune:true — this Kustomization solely owns the VM manifests (no adoption concern).
|
||||
# dependsOn platform-config so KubeVirt/CDI + the longhorn-vm StorageClass exist first, AND
|
||||
# platform-config-cdi so the CDI StorageProfiles do (2026-09-05: the StorageProfiles MOVED OUT of
|
||||
# platform-config into their own tier, so `dependsOn: platform-config` alone NO LONGER guarantees
|
||||
# them — see the dependsOn block). prune:true — this Kustomization solely owns the VM manifests (no
|
||||
# adoption concern).
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
@@ -19,6 +22,28 @@ spec:
|
||||
wait: false
|
||||
dependsOn:
|
||||
- name: platform-config
|
||||
# LOAD-BEARING, not decoration. apps/vms/unifi/20-datavolume.yaml provisions a DataVolume on
|
||||
# storageClassName `longhorn-vm`, and CDI reads StorageProfile/longhorn-vm AT PROVISIONING TIME to
|
||||
# decide the claim's accessModes/volumeMode. Without that profile's `claimPropertySets`
|
||||
# (ReadWriteMany + Block) the PVC is stamped RWO/Filesystem and the VM disk is PERMANENTLY not
|
||||
# live-migratable — a silent, unrecoverable-without-recreate loss. The profiles used to arrive
|
||||
# with platform-config; since 2026-09-05 they arrive with `platform-config-cdi`, so the edge has
|
||||
# to move with them. platform-config-cdi runs wait:true and StorageProfile has no status
|
||||
# conditions (kstatus -> Current on apply), so this edge cannot stall this tier.
|
||||
#
|
||||
# HONEST LIMIT: CDI provisions from the profile's STATUS, which its storageprofile-controller
|
||||
# derives from our spec. This edge proves the spec is applied, not that CDI has folded it into
|
||||
# status — that is a watch-triggered round-trip of seconds against a ~30s dependency requeue, so
|
||||
# it is a timing argument rather than a guarantee. See clusters/base/platform-config-cdi.yaml.
|
||||
#
|
||||
# This edge also makes the StorageProfile prune/recreate transition safe: platform-config's prune
|
||||
# deletes the two profiles for the ~30-60s before platform-config-cdi re-applies them, and this
|
||||
# dependsOn is what guarantees Flux cannot create a DataVolume inside that window.
|
||||
#
|
||||
# The platform-config edge above is kept although platform-config-cdi already dependsOn it: the
|
||||
# edges are a set, there is no cycle, and stating both means a future edit to platform-config-cdi
|
||||
# cannot silently drop this tier's KubeVirt/StorageClass gate.
|
||||
- name: platform-config-cdi
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
|
||||
@@ -14,6 +14,15 @@ resources:
|
||||
- cilium.yaml # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
|
||||
- platform.yaml # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
|
||||
- platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
|
||||
# SECOND-ORDER CDI configs — CRs whose CRD is created by a controller AT RUNTIME, not by any
|
||||
# manifest in git. cdi-operator.yaml registers only `cdis.cdi.kubevirt.io`; `storageprofiles` (and
|
||||
# datavolumes, datasources, …) appear only after cdi-operator reconciles the `CDI` CR that
|
||||
# platform-config applies. Because Flux dry-runs every object in a pass before applying any of them,
|
||||
# keeping the StorageProfiles in platform-config was a TRUE DEADLOCK (unknown kind -> pass aborts ->
|
||||
# CR never applied -> CRD never created -> tier never converges), invisible on dev and fatal on any
|
||||
# cold cluster — observed on prod 2026-09-05. This tier is the durable fix; `standup.sh crds` stays
|
||||
# as the belt-and-braces repair for clusters already wedged. See clusters/base/platform-config-cdi.yaml.
|
||||
- platform-config-cdi.yaml # Flux Kustomization -> ./platform-config-cdi (dependsOn platform-config)
|
||||
# App tier, split into dependency-ordered WAVES (was the single ./apps `apps` Kustomization).
|
||||
# Each apps-wave-<N> dependsOn wave N-1 (wave 0 dependsOn platform-config) and health-gates
|
||||
# (wait:true) ONLY its own wave, so a crashloop in a later wave can't stall an earlier one.
|
||||
|
||||
@@ -0,0 +1,135 @@
|
||||
---
|
||||
# Flux Kustomization: SECOND-ORDER CDI CONFIGS — reconciles ./platform-config-cdi.
|
||||
# Serves R-GITOPS-01 (all desired state in git, reconciled by Flux) and R-ENV-02.
|
||||
#
|
||||
# WHY THIS TIER EXISTS — the durable fix for the 2026-09-05 prod CDI DEADLOCK
|
||||
#
|
||||
# `platform-config` applied the `CDI` CR and the CDI `StorageProfile`s in the SAME pass.
|
||||
# kustomize-controller server-side DRY-RUNS EVERY object before applying ANY of them, and
|
||||
# `storageprofiles.cdi.kubevirt.io` does not exist until cdi-operator has reconciled that `CDI` CR.
|
||||
# So the unknown kind aborted the whole tier → the CR was never applied → the CRD was never created →
|
||||
# platform-config could NEVER converge and every tier below it stayed blocked. Not a race: a
|
||||
# deadlock. cdi-operator was Available for 12h on prod with no CR. Invisible on dev (its CDI CRDs
|
||||
# have existed for months), FATAL on any cold cluster. See DECISIONS-LOG 2026-09-05.
|
||||
#
|
||||
# `bootstrap/standup.sh crds` pre-applies the `CDI` CR to break it by hand. That is a WORKAROUND and
|
||||
# stays as belt-and-braces for clusters already wedged; THIS Kustomization is the fix that makes the
|
||||
# graph order itself, so a cold cluster needs no human in the loop.
|
||||
#
|
||||
# THE ORDERING GUARANTEE, precisely: `dependsOn` is REVISION-gated, not merely Ready-gated (verified
|
||||
# against kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies`, our
|
||||
# version): sharing a GitRepository with the dependency, this Kustomization cannot reconcile revision
|
||||
# N until `platform-config` has APPLIED revision N — and, because platform-config runs `wait: true`,
|
||||
# not until the `CDI` CR it applied is HEALTHY. cdi-operator creates the remaining CDI CRDs as part of
|
||||
# reaching that state, so by the time this tier's first dry-run runs, `StorageProfile` is a known
|
||||
# kind. One edge, no pre-apply.
|
||||
#
|
||||
# WHY NOT just move the CDI CR earlier, into `platform`? Because `platform` is the tier the CNI gates
|
||||
# and everything else waits on: putting CDI's health on that critical path is the mistake the
|
||||
# oauth2-proxy/headlamp splits were made to undo. The CR is correctly a platform-config CR; it is the
|
||||
# StorageProfiles that were in the wrong pass.
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: platform-config-cdi
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 10m
|
||||
# 2m (short, deliberately). kustomize-controller caches API discovery, so on a COLD cluster the
|
||||
# very first attempt here can still fail `no matches for kind "StorageProfile"` even though the CRD
|
||||
# now exists — a stale RESTMapper, not a missing CRD (the failure mode `standup.sh crds` warns
|
||||
# about). The controller resets the mapper and the retry succeeds, so the only cost of that stale
|
||||
# cache is one retryInterval. Keep it short.
|
||||
retryInterval: 2m
|
||||
# 2m is ample: this tier applies two tiny cluster-scoped CRs. There is no workload to roll out, no
|
||||
# image to pull and no controller to become Available, so a slow apply here means a real error, not
|
||||
# a slow rollout — failing fast surfaces it instead of burning 5m per attempt.
|
||||
timeout: 2m
|
||||
# TRUE, and this tier is the SOLE declarer of both StorageProfiles (checked repo-wide), so prune is
|
||||
# component-scoped: the stale set can only ever be a StorageProfile this file stopped rendering.
|
||||
# It is NOT the platform/cilium case — nothing here declares a shared Namespace, or any object a
|
||||
# second path also declares. See the transition note at the bottom of this file.
|
||||
prune: true
|
||||
# TRUE, but understand exactly what it buys and what it does NOT.
|
||||
#
|
||||
# CANNOT STALL: `StorageProfile`'s status type (CDI v1beta1 StorageProfileStatus: storageClass,
|
||||
# provisioner, claimPropertySets, cloneStrategy, dataImportCronSourceFormat) carries NO
|
||||
# `conditions` and NO `observedGeneration`, so kstatus's generic path reports it Current the moment
|
||||
# it is applied. `wait: true` therefore adds a Ready condition without adding a way to hang — which
|
||||
# is why it is safe to let `apps-vms` gate on it (contrast the headlamp gating stall).
|
||||
# If a resource that CAN be unhealthy is ever added to ./platform-config-cdi, re-derive this:
|
||||
# `apps-vms` waits on it.
|
||||
#
|
||||
# WHAT IT DOES NOT PROVE (honest gap): CDI reads the profile's **status**, not its spec, when it
|
||||
# provisions a claim (status.claimPropertySets / status.cloneStrategy), and CDI's
|
||||
# storageprofile-controller computes that status from our spec override plus inference. So Ready
|
||||
# here means "the spec override is accepted by the API server", not "CDI has folded it into
|
||||
# status". That reconcile is a watch-triggered round-trip of seconds, and the dependency requeue
|
||||
# below it is ~30s, so a DataVolume cannot realistically beat it — but it is a timing argument, not
|
||||
# a guarantee, and no `wait` setting can turn it into one (there is no condition to wait on).
|
||||
wait: true
|
||||
dependsOn:
|
||||
# platform-config applies the `CDI` CR and (wait:true) waits for it to be healthy — which is what
|
||||
# causes cdi-operator to create `storageprofiles.cdi.kubevirt.io`. Revision-gated, so this is an
|
||||
# ordering guarantee and not a hope. Transitively this also chains platform -> cilium ->
|
||||
# flux-system, so `cluster-vars` at revision N is applied before this renders revision N.
|
||||
- name: platform-config
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./platform-config-cdi
|
||||
# NO SOPS `decryption` stanza: ../platform/kubevirt/cdi-storageprofiles.yaml is two plain CRs with
|
||||
# no Secret and no *.sops.yaml, so neither overlay needs to patch one in (unlike
|
||||
# oauth2-proxy/headlamp/crafty/velero, which each carry an encrypted Secret). Nothing to add to
|
||||
# clusters/dev or clusters/prod for this tier.
|
||||
#
|
||||
# substituteFrom cluster-vars: the manifest currently contains NO `${TOKEN}`, so this stanza
|
||||
# substitutes nothing today. It is declared anyway for two concrete reasons, not for symmetry:
|
||||
# 1. COVERAGE. conformance/tests/l0-static/flux-postbuild-substitution/check.sh only audits paths
|
||||
# whose Kustomization declares substituteFrom. Omitting it would move
|
||||
# cdi-storageprofiles.yaml OUT of the audited set — a blind spot where a future `${TOKEN}`
|
||||
# could ship as a literal string, precisely the class of bug that check exists to stop.
|
||||
# 2. FAIL-CLOSED. kustomize-controller runs --feature-gates=StrictPostBuildSubstitutions=true, so
|
||||
# with this stanza an undefined token FAILS this Kustomization loudly instead of rendering
|
||||
# empty. Without the stanza there is no substitution at all and the literal ships silently.
|
||||
postBuild:
|
||||
substituteFrom:
|
||||
- kind: ConfigMap
|
||||
name: cluster-vars
|
||||
# ── FIRST RECONCILE AFTER MERGE: a TRANSIENT DELETE-THEN-RECREATE. READ BEFORE MERGING. ──────────
|
||||
# Moving cdi-storageprofiles.yaml out of ./platform-config drops both StorageProfiles from THAT
|
||||
# Kustomization's inventory, and platform-config has `prune: true` — so its next apply DELETES
|
||||
# StorageProfile/longhorn and StorageProfile/longhorn-vm, and this Kustomization recreates them on
|
||||
# the following reconcile (revision-gated: strictly after platform-config's apply, so there is no
|
||||
# dual-ownership fight — the objects are pruned before they are re-applied under this tier's
|
||||
# `kustomize.toolkit.fluxcd.io/name=platform-config-cdi` label).
|
||||
#
|
||||
# Why that is acceptable here — and it is NOT the platform/cilium case:
|
||||
# * Neither object is the sole (or any) declarer of a shared object. No Namespace, no ConfigMap,
|
||||
# nothing else in the repo declares a StorageProfile. The stale set is exactly two
|
||||
# cluster-scoped CDI CRs.
|
||||
# * The objects do not actually go missing: CDI's storageprofile-controller creates a
|
||||
# StorageProfile per StorageClass and recreates a deleted one, with an EMPTY `.spec` and
|
||||
# inferred `.status`. So the transient is "our .spec overrides are absent", not "the object is
|
||||
# absent".
|
||||
# * A StorageProfile is read at DataVolume/PVC PROVISIONING and CLONE time only. Existing PVCs,
|
||||
# existing DataVolumes and running VMs do not consult it again, so nothing already provisioned
|
||||
# is disturbed and no data path is touched.
|
||||
# * The only exposure is a DataVolume/clone STARTED inside the window: it would clone by `copy`
|
||||
# instead of `snapshot`, and on `longhorn-vm` be stamped RWO/Filesystem instead of RWX/Block —
|
||||
# permanently non-live-migratable for that disk. Flux cannot cause that, because `apps-vms` now
|
||||
# dependsOn THIS tier (clusters/base/apps-vms.yaml), so no Flux-managed DataVolume can be
|
||||
# created while the overrides are absent. A human running `virtctl`/`kubectl` in that minute
|
||||
# could.
|
||||
#
|
||||
# PRECONDITION THE OPERATOR MUST CHECK, because it is what bounds the window: the recreate cannot
|
||||
# happen until `platform-config` is Ready, and platform-config runs `wait: true` over the whole
|
||||
# config tier (ClusterIssuers, wildcard Certificate, Gateway, Keycloak, CNPG, KubeVirt, CDI…). On a
|
||||
# healthy cluster that is seconds and the window is ~30-60s (kustomize-controller's
|
||||
# --requeue-dependency default is 30s). If platform-config is NOT Ready when this merges, the
|
||||
# StorageProfiles are pruned and stay at CDI defaults until it recovers. So:
|
||||
# `flux get kustomization platform-config` must be Ready BEFORE merging.
|
||||
# If it is not, do the two-step the cilium move used instead: merge with `platform-config.prune`
|
||||
# temporarily set to `false` (the objects are then dropped from its inventory but NOT deleted, and
|
||||
# this tier ADOPTS them with zero window), verify the ownership label transferred, then restore
|
||||
# `prune: true` in a second commit.
|
||||
@@ -0,0 +1,41 @@
|
||||
---
|
||||
# platform-config-cdi — the SECOND-ORDER config wave: custom resources whose CRD is created at
|
||||
# RUNTIME by a controller that a `platform-config` custom resource switches on.
|
||||
#
|
||||
# ── THE BUG THIS DIRECTORY EXISTS TO FIX (observed on PROD, 2026-09-05) ───────────────────────────
|
||||
# The repo already splits CONTROLLERS (`platform`) from CONFIGS (`platform-config`) because a CR
|
||||
# cannot be dry-run before its CRD is registered. That split assumes a CRD is registered by the
|
||||
# controller INSTALL. CDI breaks the assumption: `cdi-operator.yaml` registers exactly ONE CRD
|
||||
# (`cdis.cdi.kubevirt.io`). Every OTHER CDI CRD — `storageprofiles`, `datavolumes`, `datasources`,
|
||||
# `dataimportcrons`, … — is created by cdi-operator only AFTER it has reconciled the `CDI` CR.
|
||||
#
|
||||
# So `cdi-cr.yaml` is a FIRST-order CR (its CRD comes from the install) and belongs in
|
||||
# platform-config, while `cdi-storageprofiles.yaml` is a SECOND-order CR (its CRD comes from
|
||||
# reconciling the first-order CR). Having both in ONE Flux Kustomization is a true DEADLOCK, not a
|
||||
# race: kustomize-controller server-side dry-runs EVERY object before applying ANY of them, so the
|
||||
# unknown `StorageProfile` kind aborts the whole pass → the `CDI` CR is never applied → the
|
||||
# StorageProfile CRD is never created → platform-config can NEVER converge, and every tier that
|
||||
# dependsOn it stays blocked forever. On prod, cdi-operator sat Available for 12h with no CR:
|
||||
#
|
||||
# StorageProfile/longhorn dry-run failed: no matches for kind "StorageProfile"
|
||||
# in version "cdi.kubevirt.io/v1beta1"
|
||||
#
|
||||
# Invisible on dev (whose CDI CRDs have existed for months) and fatal on any COLD cluster.
|
||||
#
|
||||
# ── THE RULE, so this is not rediscovered a third time ────────────────────────────────────────────
|
||||
# A CR belongs HERE (not in platform-config) when the CRD it needs is created by a CONTROLLER AT
|
||||
# RUNTIME rather than by a manifest in git. Grep test: if `grep -rn "kind: CustomResourceDefinition"`
|
||||
# over ./platform (following remote bases) does not produce the CRD, and no chart installs it, then
|
||||
# some operator creates it after being switched on — and this is the tier for it.
|
||||
#
|
||||
# Files are referenced IN PLACE via ../platform/... (Flux builds with LoadRestrictionsNone, and the
|
||||
# manifests stay beside the component they configure). Build/verify locally with:
|
||||
# kubectl kustomize --load-restrictor LoadRestrictionsNone platform-config-cdi
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
# CDI StorageProfiles — snapshot smart-clone for `longhorn`, and RWX/Block + snapshot clone for the
|
||||
# `longhorn-vm` virt-default class (clears CDIDefaultStorageClassDegraded). `StorageProfile`'s CRD
|
||||
# is created by cdi-operator only once the `CDI` CR (applied by platform-config) is reconciled,
|
||||
# which is exactly why these cannot live in the same pass as that CR.
|
||||
- ../platform/kubevirt/cdi-storageprofiles.yaml
|
||||
@@ -41,9 +41,23 @@ resources:
|
||||
# ConfigMap kept beside them for locality.
|
||||
- ../platform/kubevirt/kubevirt-cr.yaml
|
||||
- ../platform/kubevirt/cdi-cr.yaml
|
||||
# CDI StorageProfiles: snapshot smart-clone for `longhorn`, and RWX/Block + snapshot clone for the
|
||||
# `longhorn-vm` virt-default class (consume cdi.kubevirt.io; clears CDIDefaultStorageClassDegraded).
|
||||
- ../platform/kubevirt/cdi-storageprofiles.yaml
|
||||
# *** DO NOT ADD ../platform/kubevirt/cdi-storageprofiles.yaml BACK HERE. IT DEADLOCKS THIS TIER. ***
|
||||
# The CDI StorageProfiles were MOVED OUT on 2026-09-05 into their own Flux Kustomization
|
||||
# (clusters/base/platform-config-cdi.yaml -> ./platform-config-cdi, dependsOn platform-config).
|
||||
#
|
||||
# `StorageProfile`'s CRD is NOT registered by the CDI operator INSTALL — cdi-operator.yaml ships only
|
||||
# `cdis.cdi.kubevirt.io`. cdi-operator creates `storageprofiles` (and datavolumes, datasources, …)
|
||||
# only AFTER it has reconciled the `CDI` CR two lines above. Flux server-side DRY-RUNS EVERY object
|
||||
# in a Kustomization before applying ANY of them, so having both in this list meant: unknown kind
|
||||
# `StorageProfile` -> the whole pass aborts -> the `CDI` CR is never applied -> the CRD is never
|
||||
# created -> THIS TIER CAN NEVER CONVERGE, and every tier that dependsOn it is blocked forever.
|
||||
# A true deadlock, not a race. Observed on prod 2026-09-05 (cdi-operator Available 12h, no CR):
|
||||
# StorageProfile/longhorn dry-run failed: no matches for kind "StorageProfile" in version
|
||||
# "cdi.kubevirt.io/v1beta1"
|
||||
# Invisible on dev, whose CDI CRDs have existed for months. See DECISIONS-LOG 2026-09-05.
|
||||
#
|
||||
# THE RULE: a CR whose CRD is created by a controller AT RUNTIME (rather than by a manifest in git)
|
||||
# is SECOND-ORDER and belongs in ./platform-config-cdi, not here.
|
||||
- ../platform/kubevirt/servicemonitor.yaml
|
||||
- ../platform/kubevirt/alerts.yaml
|
||||
- ../platform/kubevirt/dashboard.yaml
|
||||
|
||||
@@ -20,8 +20,12 @@ Neither k3s nor the Longhorn chart ships this. It is a prerequisite for:
|
||||
## Wiring
|
||||
|
||||
- **Pass:** `platform` (controllers). Registers the CRDs.
|
||||
- The `VolumeSnapshotClass` / `StorageProfile` CRs that consume the CRDs live in the
|
||||
`platform-config` pass (`dependsOn: platform`), per the controllers/configs split.
|
||||
- The `VolumeSnapshotClass` CR that consumes the CRDs lives in the `platform-config` pass
|
||||
(`dependsOn: platform`), per the controllers/configs split.
|
||||
- The CDI `StorageProfile`s live one pass further out, in `platform-config-cdi`
|
||||
(`dependsOn: platform-config`) — `storageprofiles.cdi.kubevirt.io` is created by cdi-operator only
|
||||
after it reconciles the `CDI` CR, so they cannot share a pass with it. See
|
||||
`clusters/base/platform-config-cdi.yaml`.
|
||||
|
||||
## Version
|
||||
|
||||
|
||||
@@ -9,8 +9,10 @@
|
||||
# * no VolumeSnapshotClass can be created (item 2), and
|
||||
# * CDI cannot do SNAPSHOT smart-clone (item 3) — it falls back to slow host-assisted `copy`,
|
||||
# which leaves the default virt StorageClass "degraded" (CDIDefaultStorageClassDegraded).
|
||||
# Installing it here (controllers pass) registers the CRDs; the VolumeSnapshotClass + the CDI
|
||||
# StorageProfiles that CONSUME them are applied by platform-config (dependsOn: platform).
|
||||
# Installing it here (controllers pass) registers the CRDs; the VolumeSnapshotClass that CONSUMES them
|
||||
# is applied by platform-config (dependsOn: platform), and the CDI StorageProfiles that set
|
||||
# `cloneStrategy: snapshot` by platform-config-cdi (dependsOn: platform-config — their own CRD is
|
||||
# created by cdi-operator only after the `CDI` CR reconciles; see clusters/base/platform-config-cdi.yaml).
|
||||
#
|
||||
# CONTROLLERS/CONFIGS SPLIT (F-MON1): this is the CONTROLLERS half. The snapshot.storage.k8s.io
|
||||
# CRDs are registered by the `client/config/crd` base below; the snapshot-controller Deployment
|
||||
|
||||
@@ -1,9 +1,19 @@
|
||||
---
|
||||
# CDI StorageProfiles — the storage "recipes" CDI uses when it provisions/clones VM disk PVCs
|
||||
# (ADR-0005). CONSUME the cdi.kubevirt.io CRD registered by the CDI operator (platform pass), so —
|
||||
# like cdi-cr.yaml — these CRs are applied by the `platform-config` Flux Kustomization
|
||||
# (dependsOn: platform). They live here beside the KubeVirt/CDI component and are referenced in
|
||||
# place from platform-config/kustomization.yaml.
|
||||
# (ADR-0005).
|
||||
#
|
||||
# APPLIED BY `platform-config-cdi`, NOT `platform-config` (moved 2026-09-05). These are SECOND-ORDER
|
||||
# CRs: `storageprofiles.cdi.kubevirt.io` is NOT registered by the CDI operator install (cdi-operator.yaml
|
||||
# ships only `cdis.cdi.kubevirt.io`) — cdi-operator creates it only AFTER it has reconciled the `CDI` CR
|
||||
# that platform-config applies. Flux server-side dry-runs every object in a pass before applying any of
|
||||
# them, so co-locating these with that CR was a TRUE DEADLOCK: unknown kind -> pass aborts -> CR never
|
||||
# applied -> CRD never created -> platform-config never converges -> every tier below it blocked. It was
|
||||
# invisible on dev (CDI CRDs long present) and cost 12h on prod's cold start, 2026-09-05.
|
||||
# They live here beside the KubeVirt/CDI component and are referenced in place from
|
||||
# platform-config-cdi/kustomization.yaml (dependsOn: platform-config, revision-gated).
|
||||
#
|
||||
# `apps-vms` dependsOn platform-config-cdi for these: a DataVolume created before the longhorn-vm
|
||||
# profile below exists is stamped RWO/Filesystem and is permanently not live-migratable.
|
||||
#
|
||||
# CDI AUTO-CREATES a StorageProfile per StorageClass and fills its STATUS (inferred capabilities).
|
||||
# The operator owns `.status`; the USER owns `.spec`. Setting `.spec` here does NOT fight CDI — it
|
||||
|
||||
@@ -8,6 +8,16 @@
|
||||
# (dependsOn: platform, wait) so the CRDs exist BEFORE the CRs apply. Applying both in one atomic pass
|
||||
# deadlocks ("no matches for kind KubeVirt"), exactly the failure F-MON1 documents.
|
||||
#
|
||||
# THERE IS A THIRD PASS FOR CDI, and it is not optional (2026-09-05). The two operator manifests below
|
||||
# register exactly ONE CDI CRD, `cdis.cdi.kubevirt.io`. Every other CDI CRD — `storageprofiles`,
|
||||
# `datavolumes`, `datasources`, `dataimportcrons` — is created by cdi-operator only AFTER it reconciles
|
||||
# the `CDI` CR. So `cdi-storageprofiles.yaml` is a SECOND-ORDER CR and CANNOT share a pass with
|
||||
# `cdi-cr.yaml`: the dry-run of the unknown `StorageProfile` kind aborts the pass, the CR is never
|
||||
# applied, the CRD is never created, and platform-config deadlocks (cost 12h on prod's cold start).
|
||||
# It is therefore applied by a third Kustomization, `platform-config-cdi` (dependsOn platform-config).
|
||||
# Before adding any new CDI CR here, ask which pass owns it: CRD in git -> platform-config;
|
||||
# CRD created by cdi-operator at runtime -> platform-config-cdi.
|
||||
#
|
||||
# Namespaces: the operator manifests each create their own namespace (`kubevirt`, `cdi`) with the
|
||||
# pod-security profile they need (kubevirt is privileged) — so we do NOT redeclare them here (that
|
||||
# would be a duplicate-resource kustomize error). These are PLATFORM namespaces: they intentionally
|
||||
|
||||
Reference in new issue
Block a user