arXiv:2502.10458cs.LGcs.AI2025-02ICML被引 19

让文生图模型具备多模态上下文推理能力,只需少量训练即可显著提升生成逻辑性。

I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models

论文配图:I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models
图 1 · 摘自论文原文
  • 通过视觉语言模型与大语言模型解码器对齐,实现文生图模型的上下文理解。
  • 在CoBSAT测试中推理准确率从19.2%提升至46.3%,仅需5小时训练。
  • 适合需要逻辑连贯图文生成的应用,如创意设计、智能内容创作。

本文提出ThinkDiff,一种新型对齐范式,通过融合视觉语言模型(VLM)的优势,赋予文生图扩散模型多模态上下文理解与推理能力。现有方法多关注像素级重建,受限于复杂且稀缺的推理数据集。ThinkDiff采用视觉语言训练作为代理任务,将VLM与编码器-解码器架构的大语言模型(LLM)解码器对齐,而非直接对齐扩散解码器。该方法基于观察:LLM解码器与使用对应LLM编码器进行提示嵌入的扩散解码器共享相同的输入特征空间。因此,通过与LLM解码器对齐,可简化对扩散解码器的对齐过程。无需复杂训练与数据集,ThinkDiff有效激活了扩散模型的理解、推理与组合能力。实验表明,在挑战性的CoBSAT多模态上下文推理生成基准上,准确率从19.2%提升至46.3%,仅需在4块A100 GPU上训练5小时。此外,该方法在组合多个图像与文本生成逻辑一致图像方面表现优异。

原文摘要 · Abstract (English)

This paper presents ThinkDiff, a novel alignment paradigm that empowers text-to-image diffusion models with multimodal in-context understanding and reasoning capabilities by integrating the strengths of vision-language models (VLMs). Existing multimodal diffusion finetuning methods largely focus on pixel-level reconstruction rather than in-context reasoning, and are constrained by the complexity and limited availability of reasoning-based datasets. ThinkDiff addresses these challenges by leveraging vision-language training as a proxy task, aligning VLMs with the decoder of an encoder-decoder large language model (LLM) instead of a diffusion decoder. This proxy task builds on the observation that the $\textbf{LLM decoder}$ shares the same input feature space with $\textbf{diffusion decoders}$ that use the corresponding $\textbf{LLM encoder}$ for prompt embedding. As a result, aligning VLMs with diffusion decoders can be simplified through alignment with the LLM decoder. Without complex training and datasets, ThinkDiff effectively unleashes understanding, reasoning, and composing capabilities in diffusion models. Experiments demonstrate that ThinkDiff significantly improves accuracy from 19.2% to 46.3% on the challenging CoBSAT benchmark for multimodal in-context reasoning generation, with only 5 hours of training on 4 A100 GPUs. Additionally, ThinkDiff demonstrates exceptional performance in composing multiple images and texts into logically coherent images. Project page: https://mizhenxing.github.io/ThinkDiff.

文生图多模态推理扩散模型对齐学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。