用大模型直接根据文本生成3D布局,让室内场景更灵活真实。
Direct Numerical Layout Generation for 3D Indoor Scene Synthesis via Spatial Reasoning
- 通过分步推理直接从文本生成三维布局
- 在多个数据集上实现高语义一致性与物理合理性
- 适合需要精准控制的虚拟场景设计和智能体应用
真实感3D室内场景合成对具身智能和数字内容创作至关重要,可自然分为物体生成与布局生成两部分。尽管生成模型在物体层面已取得显著进展,但布局生成因数据集有限而仍具挑战:现有方法或过度依赖数据、或依赖预设约束,牺牲灵活性,难以实现开放词汇与细粒度指令对齐。本文提出DirectLayout框架,利用大语言模型(LLM)的通用空间推理能力,直接从文本描述生成数值型3D布局。该框架分三阶段进行:生成鸟瞰图(BEV)布局、将其升维至3D空间、优化物体位置。为增强空间推理能力,基于3D-Front数据集引入基于思维链(CoT)的激活机制,并设计了CoT-Grounded Generative Layout Reward以提升泛化性与规划能力。推理时,通过上下文学习实现迭代资产-布局对齐,解决资产与布局不匹配问题。大量实验表明,DirectLayout在语义一致性、泛化性和物理合理性方面表现优异。
原文摘要 · Abstract (English)
Realistic 3D indoor scene synthesis is vital for embodied AI and digital content creation. It can be naturally divided into two subtasks: object generation and layout generation. While recent generative models have significantly advanced object-level quality and controllability, layout generation remains challenging due to limited datasets. Existing methods either overfit to these datasets or rely on predefined constraints to optimize numerical layout that sacrifice flexibility. As a result, they fail to generate scenes that are both open-vocabulary and aligned with fine-grained user instructions. We introduce DirectLayout, a framework that directly generates numerical 3D layouts from text descriptions using generalizable spatial reasoning of large language models (LLMs). DirectLayout decomposes the generation into three stages: producing a Bird's-Eye View (BEV) layout, lifting it into 3D space, and refining object placements. To enable explicit spatial reasoning and help the model grasp basic principles of object placement, we employ Chain-of-Thought (CoT) Activation based on the 3D-Front dataset. Additionally, we design CoT-Grounded Generative Layout Reward to enhance generalization and spatial planning. During inference, DirectLayout addresses asset-layout mismatches via Iterative Asset-Layout Alignment through in-context learning. Extensive experiments demonstrate that DirectLayout achieves impressive semantic consistency, generalization and physical plausibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。