用高斯表示+自监督学习，实现无需标注的3D语义场景理解
GaussTR: Foundation Model-Aligned Gaussian Transformer for Self-Supervised 3D Spatial Understanding
- 用稀疏高斯点集建模3D场景，前向传播预测
- 零样本下在Occ3D-nuScenes上达12.27 mIoU，训练快40%
- 适合自动驾驶与智能体场景理解，支持开放词汇
3D语义占据预测是空间理解的基础，但现有方法因依赖大量标注数据和计算密集的体素表示，在可扩展性和泛化能力上面临挑战。本文提出GaussTR，一种基于高斯的Transformer框架，通过高斯表示统一稀疏3D建模与基础模型对齐，推动3D空间理解发展。GaussTR以前向方式预测3D场景中的稀疏高斯集合，并将高斯投影至2D视图，使渲染特征与基础模型对齐，实现自监督3D表征学习，无需显式标注即可完成开放词汇语义占据预测。在Occ3D-nuScenes数据集上的实验证明，GaussTR在零样本条件下达到12.27 mIoU的领先性能，训练时间减少40%。结果表明，GaussTR在可扩展、全貌的3D空间理解中表现优异，对自动驾驶与具身智能体具有重要意义。代码已开源：https://github.com/hustvl/GaussTR。
原文摘要 · Abstract (English)
3D Semantic Occupancy Prediction is fundamental for spatial understanding, yet existing approaches face challenges in scalability and generalization due to their reliance on extensive labeled data and computationally intensive voxel-wise representations. In this paper, we introduce GaussTR, a novel Gaussian-based Transformer framework that unifies sparse 3D modeling with foundation model alignment through Gaussian representations to advance 3D spatial understanding. GaussTR predicts sparse sets of Gaussians in a feed-forward manner to represent 3D scenes. By splatting the Gaussians into 2D views and aligning the rendered features with foundation models, GaussTR facilitates self-supervised 3D representation learning and enables open-vocabulary semantic occupancy prediction without requiring explicit annotations. Empirical experiments on the Occ3D-nuScenes dataset demonstrate GaussTR's state-of-the-art zero-shot performance of 12.27 mIoU, along with a 40% reduction in training time. These results highlight the efficacy of GaussTR for scalable and holistic 3D spatial understanding, with promising implications in autonomous driving and embodied agents. The code is available at https://github.com/hustvl/GaussTR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。