arXiv:2601.10332cs.CV2026-01被引 13

让AI先理解再生成,提升图文一致性。

Think-Then-Generate: Reasoning-Aware Text-to-Image Diffusion with LLM Encoders

  • 用LLM推理重写提示词,再生成图像
  • 在WISE评分达0.79,接近GPT-4水平
  • 适合需要精准语义表达的图像生成任务

近期文本到图像扩散模型在多样文本提示下实现了高质量视觉生成。然而,多数现有模型即使使用大语言模型(LLM)作为文本编码器,仍仅将文本当作像素映射——未利用其推理能力推断应呈现的内容。为此,我们提出“思考-生成”(T2G)范式:通过激活LLM编码器的思考与重写能力,将重构后的提示状态作为扩散模型的条件。首先采用轻量级监督微调激活该模式;随后通过双GRPO对齐优化编码器与扩散主干,确保上下文推理准确、语义还原精确。其中,文本编码器受图像锚定奖励激励,以推断并召回世界知识;扩散主干则被驱动生成语义一致且视觉连贯的图像。实验表明,在基于推理的图像生成与编辑基准上,事实一致性、语义对齐和视觉真实感均显著提升,WISE得分达0.79,接近GPT-4水平。结果为具备推理、表达与演示能力的下一代统一模型迈出关键一步。

原文摘要 · Abstract (English)

Recent progress in text-to-image (T2I) diffusion models (DMs) has enabled high-quality visual synthesis from diverse textual prompts. Yet, most existing T2I DMs, even those equipped with large language model (LLM)-based text encoders, remain text-pixel mappers -- they employ LLMs merely as text encoders, without leveraging their inherent reasoning capabilities to infer what should be visually depicted given the textual prompt. To move beyond such literal generation, we propose the think-then-generate (T2G) paradigm, where the LLM-based text encoder is encouraged to reason about and rewrite raw user prompts; the states of the rewritten prompts then serve as diffusion conditioning. To achieve this, we first activate the think-then-rewrite pattern of the LLM encoder with a lightweight supervised fine-tuning process. Subsequently, the LLM encoder and diffusion backbone are co-optimized to ensure faithful reasoning about the context and accurate rendering of the semantics via Dual-GRPO. In particular, the text encoder is reinforced using image-grounded rewards to infer and recall world knowledge, while the diffusion backbone is pushed to produce semantically consistent and visually coherent images. Experiments show substantial improvements in factual consistency, semantic alignment, and visual realism across reasoning-based image generation and editing benchmarks, achieving 0.79 on WISE score, nearly on par with GPT-4. Our results constitute a promising step toward next-generation unified models with reasoning, expression, and demonstration capacities.

文本生成图像大模型推理扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。