让大模型直接理解3D空间,生成更合理场景布局。
NaLA: A 3D Native LLM Layout Agent for High-quality 3D Scene Generation

- 大模型直接处理3D资产和边界,避免文本转换带来的信息损失。
- 采用粗到精策略预测物体位置朝向,提升定位精度。
- 适合需要高保真3D场景生成的开发者与设计师使用。
近期大型语言模型(LLMs)在3D场景生成中展现出作为布局代理的潜力。现有布局代理仍存在布局不合理的问题,主要因为多数方法将3D资产与3D布局转化为文本描述作为输入输出,导致因模态差异造成严重信息丢失。本文提出NaLA,一种原生3D的大型语言模型布局代理,用于高质量3D场景生成。其输入直接编码3D场景边界与3D资产,保留细粒度几何信息,并支持对碰撞、表面支撑、包含等关系的显式推理。为准确输出资产的位置与朝向,NaLA采用粗到精预测机制:先以自回归方式预测离散姿态,再通过连续回归精调。在多样化布局数据集上训练后,NaLA具备强几何感知与布局一致性。实验表明,相比先前布局代理,NaLA在生成质量与推理效率方面均表现更优,且通过全面消融实验验证了各组件的有效性。
原文摘要 · Abstract (English)
Recently, Large Language Models (LLMs) have emerged as promising layout agents for 3D scene generation. Existing layout agents still suffer from implausible layout generation because most of them convert 3D assets and 3D layouts into textual descriptions as inputs and outputs, which involves severe information loss due to the modality gap between texts and 3D assets and 3D layouts. We propose NaLA, a native 3D LLM layout Agent for high-quality 3D scene generation by placing 3D assets in the scene. For the inputs, NaLA encodes 3D scene boundaries and 3D assets directly into the LLM, preserving fine-grained geometry and enabling explicit reasoning over relationships like collisions, surface supporting, and containment. To accurately output the positions and orientations of assets, NaLA adopts a coarse-to-fine prediction mechanism that first predicts discrete poses in an autoregressive manner and then refines the discrete poses with a continuous regression. Trained on diverse layout datasets, NaLA attains strong geometric perception and layout coherence. Experiments demonstrate that NaLA outperforms prior layout agents in both generation quality and inference efficiency, with comprehensive ablation studies to verify each component's effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。