用大模型当推理引擎,让图像生成更忠实于复杂提示。
Large Language Models are Universal Reasoners for Visual Generation

- 大模型先生成粗略视觉草图,再自我评估一致性。
- 改进后在复杂提示下对齐度提升,图像质量不变。
- 适合需要精准图文对齐的生成任务使用者。
文本到图像生成借助扩散模型快速进展,从CLIP和T5条件化发展到统一系统,即单一大语言模型(LLM)骨干同时处理视觉理解与生成。尽管架构统一,这些系统在合成复杂提示时仍频繁出现对齐失败,尽管其验证图像是否满足提示的能力依然高度准确。我们将其称为‘理解-生成差距’,并提出UniReasoner框架,利用LLM作为通用推理器,将理解优势转化为直接生成引导。给定提示后,LLM首先生成由离散视觉标记组成的粗略视觉草图;随后进行自评,通过评估草图与提示的一致性,生成具体指出需修正内容的基于事实的文本评价;最后,扩散模型联合条件于提示、视觉草图和评价,确保生成受到明确纠错信号的引导。每个信号弥补对方局限:草图提供场景级锚点,减少纯文本条件的模糊性;评价将验证转化为可操作的约束,纠正遗漏、幻觉和关系错误。实验表明,UniReasoner在相同扩散骨干下提升了组合对齐与语义忠实度,同时保持图像质量,展示了一种实用方法来利用大模型推理弥合理解-生成差距。
原文摘要 · Abstract (English)
Text-to-image generation has advanced rapidly with diffusion models, progressing from CLIP and T5 conditioning to unified systems where a single LLM backbone handles both visual understanding and generation. Despite the architectural unification, these systems frequently fail to faithfully align complex prompts during synthesis, even though they remain highly accurate at verifying whether an image satisfies those same prompts. We formalize this as the \emph{understanding-generation gap} and propose UniReasoner, a framework that leverages the LLM as a universal reasoner to convert its understanding strength into direct generation guidance. Given a prompt, the LLM first produces a coarse visual draft composed of discrete vision tokens. It then performs a self-critique by evaluating the draft for prompt consistency, producing a grounded textual evaluation that pinpoints what needs to be corrected. Finally, a diffusion model is conditioned jointly on the prompt, the visual draft, and the evaluation, ensuring that generation is guided by explicit corrective signals. Each signal addresses a limitation of the other: the draft provides a concrete, scene-level anchor that reduces under-specification in text-only conditioning, while the evaluation turns verification into grounded, actionable constraints that correct omissions, hallucinations, and relational errors. Experiments show that UniReasoner improves compositional alignment and semantic faithfulness under the same diffusion backbone while maintaining image quality, demonstrating a practical way to exploit LLM reasoning to close the understanding-generation gap.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。