用思维链提升文本生成图像的上下文理解能力。
ImageGen-CoT: Enhancing Text-to-Image In-context Learning with Chain-of-Thought Reasoning
- 引入ImageGen-CoT思维链，引导模型先推理再生成图像。
- 微调后在SEED-X任务上性能提升80%。
- 适合需要精准图文对齐的研究与应用。
本文研究文本到图像的上下文学习（T2I-ICL）问题。尽管统一多模态大模型（MLLMs）近年发展迅速，但在T2I-ICL场景中仍面临上下文推理能力不足的问题。为此，我们提出一种新框架，在图像生成前引入名为ImageGen-CoT的思维链过程。为避免生成无序无效的推理步骤，我们构建了自动化数据集采集流程，生成高质量的ImageGen-CoT数据集，并以此微调MLLM以增强其上下文推理能力。为进一步提升性能，探索测试时扩展策略，提出一种混合缩放方法：先生成多个ImageGen-CoT链，再对每条链通过采样生成多幅图像。大量实验表明该方法有效，尤其在SEED-X任务上，微调后性能提升达80%。项目主页见https://ImageGen-CoT.github.io/，代码与模型权重将开源。
原文摘要 · Abstract (English)
In this work, we study the problem of Text-to-Image In-Context Learning (T2I-ICL). While Unified Multimodal LLMs (MLLMs) have advanced rapidly in recent years, they struggle with contextual reasoning in T2I-ICL scenarios. To address this limitation, we propose a novel framework that incorporates a thought process called ImageGen-CoT prior to image generation. To avoid generating unstructured ineffective reasoning steps, we develop an automatic pipeline to curate a high-quality ImageGen-CoT dataset. We then fine-tune MLLMs using this dataset to enhance their contextual reasoning capabilities. To further enhance performance, we explore test-time scale-up strategies and propose a novel hybrid scaling approach. This approach first generates multiple ImageGen-CoT chains and then produces multiple images for each chain via sampling. Extensive experiments demonstrate the effectiveness of our proposed method. Notably, fine-tuning with the ImageGen-CoT dataset leads to a substantial 80\% performance gain for SEED-X on T2I-ICL tasks. See our project page at https://ImageGen-CoT.github.io/. Code and model weights will be open-sourced.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。