用3D点云训练大模型，自动识别室内结构与物体
SpatialLM: Training Large Language Models for Structured Indoor Modeling
- 基于通用多模态大模型架构，直接微调开源模型处理点云数据
- 在12,328个室内场景（54,778个房间）上训练，实现顶尖布局估计性能
- 适合增强AR、机器人等场景的空间理解能力
SpatialLM 是一个专为处理 3D 点云数据并生成结构化 3D 场景理解输出的大语言模型。其输出包含墙体、门、窗等建筑元素以及带语义类别的方向性物体框。与以往依赖特定任务网络设计的方法不同，该模型采用标准多模态大模型架构，并直接从开源大语言模型微调而来。为训练 SpatialLM，研究者构建了一个大规模高质量合成数据集，包含 12,328 个室内场景（共 54,778 个房间）的点云及其真实 3D 标注，并对多种建模与训练策略进行了系统研究。在公开基准测试中，该模型在布局估计任务上达到当前最优表现，在 3D 物体检测任务上也表现出竞争力。结果表明，该方法为提升现代大模型的空间理解能力提供了可行路径，适用于增强现实、具身机器人等领域。
原文摘要 · Abstract (English)
SpatialLM is a large language model designed to process 3D point cloud data and generate structured 3D scene understanding outputs. These outputs include architectural elements like walls, doors, windows, and oriented object boxes with their semantic categories. Unlike previous methods which exploit task-specific network designs, our model adheres to the standard multimodal LLM architecture and is fine-tuned directly from open-source LLMs. To train SpatialLM, we collect a large-scale, high-quality synthetic dataset consisting of the point clouds of 12,328 indoor scenes (54,778 rooms) with ground-truth 3D annotations, and conduct a careful study on various modeling and training decisions. On public benchmarks, our model gives state-of-the-art performance in layout estimation and competitive results in 3D object detection. With that, we show a feasible path for enhancing the spatial understanding capabilities of modern LLMs for applications in augmented reality, embodied robotics, and more.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。