用大模型+扩散模型分两阶段生成可控布局图像
A Two-Stage System for Layout-Controlled Image Generation using Large Language Models and Diffusion Models
- 先用大模型规划物体位置，再用扩散模型按布局生成图像
- 复杂场景物体召回率从57.2%提升至99.9%
- 适合需要精确布局控制的图像生成任务
文本到图像的扩散模型生成能力强大，但对物体数量和空间排列控制不足。本文提出两阶段系统解决这一组合性限制：第一阶段使用大语言模型（LLM）从物体列表生成结构化布局；第二阶段采用布局条件扩散模型生成符合该布局的逼真图像。研究发现，任务分解对基于LLM的空间规划至关重要：将初始生成简化为核心物体，并通过规则插入补全布局，使复杂场景中物体召回率从57.2%提升至99.9%。在图像合成阶段，对比两种主流条件方法——ControlNet与GLIGEN。在桌饰数据集上进行领域特定微调后，发现关键权衡：ControlNet保留基于文本的风格控制，但易产生物体幻觉；GLIGEN提供更优布局保真度，但牺牲了提示词控制能力。端到端系统成功生成指定物体数量与合理空间布局的图像，验证了解耦式方法在组合控制生成中的可行性。
原文摘要 · Abstract (English)
Text-to-image diffusion models exhibit remarkable generative capabilities, but lack precise control over object counts and spatial arrangements. This work introduces a two-stage system to address these compositional limitations. The first stage employs a Large Language Model (LLM) to generate a structured layout from a list of objects. The second stage uses a layout-conditioned diffusion model to synthesize a photorealistic image adhering to this layout. We find that task decomposition is critical for LLM-based spatial planning; by simplifying the initial generation to core objects and completing the layout with rule-based insertion, we improve object recall from 57.2% to 99.9% for complex scenes. For image synthesis, we compare two leading conditioning methods: ControlNet and GLIGEN. After domain-specific finetuning on table-setting datasets, we identify a key trade-off: ControlNet preserves text-based stylistic control but suffers from object hallucination, while GLIGEN provides superior layout fidelity at the cost of reduced prompt-based controllability. Our end-to-end system successfully generates images with specified object counts and plausible spatial arrangements, demonstrating the viability of a decoupled approach for compositionally controlled synthesis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。