用视觉语言智能体+组合扩散模型,生成符合语义和物理规则的场景布局。
LayoutAgent: A Vision-Language Agent Guided Compositional Diffusion for Spatial Layout Planning
- 引入视觉语言模型预处理输入,构建场景图并重写提示词。
- 采用机器人领域组合扩散方法生成符合关系约束的边界框布局。
- 适合需要高保真空间布局的场景生成任务,如建筑设计与虚拟拍摄。
设计真实感多对象场景不仅需要生成图像,还需规划符合语义关系和物理合理性的空间布局。现有扩散模型虽能生成高质量图像,但缺乏显式空间推理能力,导致布局不合理;而传统机器人空间规划方法虽注重几何与关系一致性,却难以捕捉视觉场景的语义丰富性。为此,本文提出LayoutAgent,一个融合视觉-语言推理与组合扩散的代理框架。给定多个含目标物体的输入图像,该方法首先通过视觉-语言模型进行分割、物体尺寸估计、场景图构建及提示词重写;随后利用传统机器人领域的组合扩散方法,生成符合场景图中物体关系的边界框布局;最后,基于前景条件的图像生成器在规划好的布局中渲染物体,生成完整场景。实验表明,LayoutAgent在布局连贯性、空间真实性和美学一致性上均优于当前最先进的布局生成模型。
原文摘要 · Abstract (English)
Designing realistic multi-object scenes requires not only generating images, but also planning spatial layouts that respect semantic relations and physical plausibility. On one hand, while recent advances in diffusion models have enabled high-quality image generation, they lack explicit spatial reasoning, leading to unrealistic object layouts. On the other hand, traditional spatial planning methods in robotics emphasize geometric and relational consistency, but they struggle to capture semantic richness in visual scenes. To bridge this gap, in this paper, we propose LayoutAgent, an agentic framework that unifies vision-language reasoning with compositional diffusion for layout generation. Given multiple input images with target objects in them, our method first employs visual-language model to preprocess the inputs through segmentation, object size estimation, scene graph construction, and prompt rewriting. Then we leverage compositional diffusion-a method traditionally used in robotics-to synthesize bounding boxes that respect object relations encoded in the scene graph for spatial layouts. In the end, a foreground-conditioned image generator composes the complete scene by rendering the objects into the planned layout guided by designed prompts. Experiments demonstrate that LayoutAgent outperforms other state-of-the-art layout generation models in layout coherence, spatial realism and aesthetic alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。