DuoGen实现通用多模态交错生成,支持图文交替创作。
DuoGen: Towards General Purpose Interleaved Multimodal Generation
- 用网页重写+合成数据构建大规模指令微调集
- 融合预训练多模态大模型与扩散生成器,提升图文一致性
- 适合需要图文协同创作的开发者与研究者
交错多模态生成超越单一模态模型能力,如分步指导、视觉规划和推理草图生成。然而现有模型在通用指令下的质量受限于训练数据不足和基础模型容量。我们提出DuoGen,一个通用交错生成框架,系统性解决数据构建、架构设计与评估问题。数据方面,通过改写精选网页内容并结合多样合成样例,构建大规模高质量指令微调数据集。架构上,利用预训练多模态大模型(MLLM)的强视觉理解能力,以及视频生成预训练扩散Transformer(DiT)的图像生成能力,避免昂贵的单模态预训练,支持灵活的基础模型选择。采用两阶段解耦策略:先指令微调MLLM,再用精心筛选的图文交错序列对齐DiT。在公开及新提出的基准上,DuoGen在文本质量、图像保真度和图像-上下文对齐方面优于已有开源模型,并在统一生成模型中达到文本到图像生成与图像编辑的顶尖水平。数据与代码将发布于 https://research.nvidia.com/labs/dir/duogen/。
原文摘要 · Abstract (English)
Interleaved multimodal generation enables capabilities beyond unimodal generation models, such as step-by-step instructional guides, visual planning, and generating visual drafts for reasoning. However, the quality of existing interleaved generation models under general instructions remains limited by insufficient training data and base model capacity. We present DuoGen, a general-purpose interleaved generation framework that systematically addresses data curation, architecture design, and evaluation. On the data side, we build a large-scale, high-quality instruction-tuning dataset by combining multimodal conversations rewritten from curated raw websites, and diverse synthetic examples covering everyday scenarios. Architecturally, DuoGen leverages the strong visual understanding of a pretrained multimodal LLM and the visual generation capabilities of a diffusion transformer (DiT) pretrained on video generation, avoiding costly unimodal pretraining and enabling flexible base model selection. A two-stage decoupled strategy first instruction-tunes the MLLM, then aligns DiT with it using curated interleaved image-text sequences. Across public and newly proposed benchmarks, DuoGen outperforms prior open-source models in text quality, image fidelity, and image-context alignment, and also achieves state-of-the-art performance on text-to-image and image editing among unified generation models. Data and code will be released at https://research.nvidia.com/labs/dir/duogen/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。