实时理解3D场景语义,支持开放词汇识别与快速重建。
EmbodiedSplat: Online Feed-Forward Semantic 3DGS for Open-Vocabulary 3D Scene Understanding
- 在线流式输入图像,实时构建带语义的3D高斯表示。
- 在300+张图像上实现端到端3D语义重建,延迟极低。
- 适合机器人导航、AR/VR等需即时环境理解的场景。
即时理解探索中的3D场景对具身任务至关重要,要求智能体在在线近乎实时条件下完成3D场景构建与理解。本文提出EmbodiedSplat,一种面向开放词汇3D场景理解的在线前馈3D高斯表示方法,可从连续输入图像中同步实现3D重建与3D语义理解。不同于以往多为离线或逐场景优化的开放词汇3DGS方法,本方法目标有二:1)从超过300张流式图像中在线重建完整场景的语义嵌入3DGS;2)通过前馈设计实现高度泛化能力,在结合实时2D模型时支持近乎实时的3D语义重建。为此,我们提出基于CLIP全局词典的在线稀疏系数场,将2D CLIP嵌入绑定至每个3D高斯点,同时控制内存开销并保持CLIP的全语义泛化性。此外,通过3D U-Net聚合3DGS部分点云生成3D几何感知的CLIP特征,弥补2D语言嵌入缺乏3D几何先验的不足。在ScanNet、ScanNet++和Replica等多个室内数据集上的大量实验验证了该方法的有效性与高效性。
原文摘要 · Abstract (English)
Understanding a 3D scene immediately with its exploration is essential for embodied tasks, where an agent must construct and comprehend the 3D scene in an online and nearly real-time manner. In this study, we propose EmbodiedSplat, an online feed-forward 3DGS for open-vocabulary scene understanding that enables simultaneous online 3D reconstruction and 3D semantic understanding from the streaming images. Unlike existing open-vocabulary 3DGS methods which are typically restricted to either offline or per-scene optimization setting, our objectives are two-fold: 1) Reconstructs the semantic-embedded 3DGS of the entire scene from over 300 streaming images in an online manner. 2) Highly generalizable to novel scenes with feed-forward design and supports nearly real-time 3D semantic reconstruction when combined with real-time 2D models. To achieve these objectives, we propose an Online Sparse Coefficients Field with a CLIP Global Codebook where it binds the 2D CLIP embeddings to each 3D Gaussian while minimizing memory consumption and preserving the full semantic generalizability of CLIP. Furthermore, we generate 3D geometric-aware CLIP features by aggregating the partial point cloud of 3DGS through 3D U-Net to compensate the 3D geometric prior to 2D-oriented language embeddings. Extensive experiments on diverse indoor datasets, including ScanNet, ScanNet++, and Replica, demonstrate both the effectiveness and efficiency of our method. Check out our project page in https://0nandon.github.io/EmbodiedSplat/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。