用公开API训练出超越先进模型的文生图模型,靠大视觉语言模型动态优化数据。
EvolveDirector: Approaching Advanced Text-to-Image Generation with Large Vision-Language Models

- 通过调用高级模型API生成海量图文对,训练基础模型。
- 仅需100万样本即逼近顶尖模型性能,远低于传统所需1000万。
- 适合想低成本复现顶尖生成能力的研究者与开发者。
近期生成模型在创造奇幻内容方面展现惊人能力,但多数依赖专有高质量数据,部分模型封闭参数仅开放API,限制下游应用。为探索利用公开资源训练媲美先进模型的文生图系统,我们提出EvolveDirector框架。该框架通过调用先进模型的公开API获取文本-图像数据对,用于训练基础模型。实验表明,基于高级模型生成的数据训练的模型可逼近其生成能力,但需超1000万样本,导致高昂的时间、算力及API调用成本。为此,我们引入预训练的大视觉语言模型(VLM)指导基础模型进化:在训练过程中,VLM持续评估模型表现,并通过判别、扩展、删除和变异操作动态更新训练集。结果表明,此范式显著降低所需数据量。此外,当同时对接多个先进模型时,EvolveDirector能选择最优生成样本,学习强大且均衡的能力。最终训练的Edgen模型表现超越这些先进模型。代码与模型权重已开源:https://github.com/showlab/EvolveDirector。
原文摘要 · Abstract (English)
Recent advancements in generation models have showcased remarkable capabilities in generating fantastic content. However, most of them are trained on proprietary high-quality data, and some models withhold their parameters and only provide accessible application programming interfaces (APIs), limiting their benefits for downstream tasks. To explore the feasibility of training a text-to-image generation model comparable to advanced models using publicly available resources, we introduce EvolveDirector. This framework interacts with advanced models through their public APIs to obtain text-image data pairs to train a base model. Our experiments with extensive data indicate that the model trained on generated data of the advanced model can approximate its generation capability. However, it requires large-scale samples of 10 million or more. This incurs significant expenses in time, computational resources, and especially the costs associated with calling fee-based APIs. To address this problem, we leverage pre-trained large vision-language models (VLMs) to guide the evolution of the base model. VLM continuously evaluates the base model during training and dynamically updates and refines the training dataset by the discrimination, expansion, deletion, and mutation operations. Experimental results show that this paradigm significantly reduces the required data volume. Furthermore, when approaching multiple advanced models, EvolveDirector can select the best samples generated by them to learn powerful and balanced abilities. The final trained model Edgen is demonstrated to outperform these advanced models. The code and model weights are available at https://github.com/showlab/EvolveDirector.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。