让机器人边移动边实时构建3D空间占据地图,精度高且效率优。
EmbodiedOcc: Embodied 3D Occupancy Prediction for Vision-based Online Scene Understanding

- 用3D高斯表示全局场景,随机器人移动逐步更新局部区域。
- 在ScanNet数据集上性能超越现有方法,实现高精度与高效预测。
- 适合需要在线感知新环境的机器人、自动驾驶等应用场景。
3D占据预测能全面描述周围场景,是3D感知的核心任务。现有方法多针对单视角或离线场景,难以适用于需通过渐进式探索理解环境的具身智能体。本文提出具身3D占据预测任务,并设计基于高斯的EmbodiedOcc框架。该框架以均匀分布的3D语义高斯初始化全局场景,随具身代理观察逐步更新局部区域。每次更新时,从观测图像中提取语义与结构特征,通过可变形交叉注意力高效融合,优化对应区域的高斯表示。最终通过高斯到体素的点阵投影生成全局3D占据图。EmbodiedOcc假设环境初始未知,以显式3D高斯形式维护全局记忆,通过局部高斯的渐进优化获取知识,符合人类具身探索认知规律。我们基于局部标注重构了EmbodiedOcc-ScanNet基准用于评估。实验表明,EmbodiedOcc显著优于现有方法,在准确率与效率上均表现优异。代码已开源:https://github.com/YkiWu/EmbodiedOcc。
原文摘要 · Abstract (English)
3D occupancy prediction provides a comprehensive description of the surrounding scenes and has become an essential task for 3D perception. Most existing methods focus on offline perception from one or a few views and cannot be applied to embodied agents that demand to gradually perceive the scene through progressive embodied exploration. In this paper, we formulate an embodied 3D occupancy prediction task to target this practical scenario and propose a Gaussian-based EmbodiedOcc framework to accomplish it. We initialize the global scene with uniform 3D semantic Gaussians and progressively update local regions observed by the embodied agent. For each update, we extract semantic and structural features from the observed image and efficiently incorporate them via deformable cross-attention to refine the regional Gaussians. Finally, we employ Gaussian-to-voxel splatting to obtain the global 3D occupancy from the updated 3D Gaussians. Our EmbodiedOcc assumes an unknown (i.e., uniformly distributed) environment and maintains an explicit global memory of it with 3D Gaussians. It gradually gains knowledge through the local refinement of regional Gaussians, which is consistent with how humans understand new scenes through embodied exploration. We reorganize an EmbodiedOcc-ScanNet benchmark based on local annotations to facilitate the evaluation of the embodied 3D occupancy prediction task. Our EmbodiedOcc outperforms existing methods by a large margin and accomplishes the embodied occupancy prediction with high accuracy and efficiency. Code: https://github.com/YkiWu/EmbodiedOcc.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。