无需训练和位姿信息，用视觉语言模型实现跨环境占位预测
FreeOcc: Training-Free Embodied Open-Vocabulary Occupancy Prediction
- 基于四层流水线构建全局一致的3D占位图，不依赖标注与学习阶段
- 在EmbodiedOcc-ScanNet上比自监督方法提升超2倍的交并比
- 零样本迁移至新环境，适合需要快速部署的机器人场景
现有基于学习的占位预测方法依赖大规模3D标注且泛化能力差。我们提出FreeOcc，一种从单目或RGB-D序列进行开放词汇占位预测的训练自由框架。与以往需体素级监督和真实相机位姿的方法不同，FreeOcc无需3D标注、位姿真值或任何学习阶段。通过四层流水线：SLAM主干估计位姿与稀疏几何；几何一致的高斯更新构建稠密3D高斯地图；利用现成的视觉-语言模型为高斯基元赋予开放词汇语义；最后通过概率性高斯到占位投影生成稠密体素占位。尽管完全无训练且位姿无关，FreeOcc在EmbodiedOcc-ScanNet上相比先前自监督方法，交并比（IoU）和平均交并比（mIoU）均提升超2倍。我们进一步引入ReplicaOcc，一个室内开放词汇占位预测基准，并证明FreeOcc可零样本迁移至新环境，显著优于监督与自监督基线。
原文摘要 · Abstract (English)
Existing learning-based occupancy prediction methods rely on large-scale 3D annotations and generalize poorly across environments. We present FreeOcc, a training-free framework for open-vocabulary occupancy prediction from monocular or RGB-D sequences. Unlike prior approaches that require voxel-level supervision and ground-truth camera poses, FreeOcc operates without 3D annotations, pose ground truth, or any learning stage. FreeOcc incrementally builds a globally consistent occupancy map via a four-layer pipeline: a SLAM backbone estimates poses and sparse geometry; a geometrically consistent Gaussian update constructs dense 3D Gaussian maps; open-vocabulary semantics from off-the-shelf vision-language models are associated with Gaussian primitives; and a probabilistic Gaussian-to-occupancy projection produces dense voxel occupancy. Despite being entirely training-free and pose-agnostic, FreeOcc achieves over $2\times$ improvements in IoU and mIoU on EmbodiedOcc-ScanNet compared to prior self-supervised methods. We further introduce ReplicaOcc, a benchmark for indoor open-vocabulary occupancy prediction, and show that FreeOcc transfers zero-shot to novel environments, substantially outperforming both supervised and self-supervised baselines. Project page: https://the-masses.github.io/freeocc-web/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。