无需训练,用预训练模型实现道路场景的全景占位预测。
FreeOcc: Training-free Panoptic Occupancy Prediction via Foundation Models
- 利用预训练模型从多视角图像中提取语义和几何信息。
- 在Occ3D-nuScenes上达到16.9 mIoU,与先进弱监督方法相当。
- 适合无标注数据、快速部署的自动驾驶场景应用。
道路场景分析中的语义与全景占位预测提供车辆周围环境的稠密三维表示。当前纯摄像头方法通常依赖昂贵的密集3D监督或需在目标域数据上训练模型,限制了在未知环境中的部署。本文提出FreeOcc,一种无需训练的流水线,利用预训练基础模型从多视角图像中恢复语义与几何信息。FreeOcc通过可提示的基础分割模型提取视图级全景先验,并结合提示到分类规则;通过重建基础模型恢复度量3D点。基于深度与置信度的过滤将可靠标签提升至3D,经时间融合与确定性精炼堆栈体素化。对于全景占位,通过拟合并合并当前视图的鲁棒3D框候选来恢复实例,实现无学习3D模型的实例感知占位。在Occ3D-nuScenes上,FreeOcc实现16.9 mIoU和16.5 RayIoU(训练免费),与最先进的弱监督方法相当。作为伪标签生成流水线用于下游模型训练时,达到21.1 RayIoU,超越此前最先进弱监督基线。此外,FreeOcc为训练免费与弱监督全景占位预测设立新基准,分别达3.1 RayPQ与3.9 RayPQ。这些结果表明,基础模型驱动的感知是实现训练免费3D场景理解的可行路径。
原文摘要 · Abstract (English)
Semantic and panoptic occupancy prediction for road scene analysis provides a dense 3D representation of the ego vehicle's surroundings. Current camera-only approaches typically rely on costly dense 3D supervision or require training models on data from the target domain, limiting deployment in unseen environments. We propose FreeOcc, a training-free pipeline that leverages pretrained foundation models to recover both semantics and geometry from multi-view images. FreeOcc extracts per-view panoptic priors with a promptable foundation segmentation model and prompt-to-taxonomy rules, and reconstructs metric 3D points with a reconstruction foundation model. Depth- and confidence- aware filtering lifts reliable labels into 3D, which are fused over time and voxelized with a deterministic refinement stack. For panoptic occupancy, instances are recovered by fitting and merging robust current-view 3D box candidates, enabling instance-aware occupancy without any learned 3D model. On Occ3D-nuScenes, FreeOcc achieves 16.9 mIoU and 16.5 RayIoU train-free, on par with state-of-the-art weakly supervised methods. When employed as a pseudo-label generation pipeline for training downstream models, it achieves 21.1 RayIoU, surpassing the previous state-of-the-art weakly supervised baseline. Furthermore, FreeOcc sets new baselines for both train-free and weakly supervised panoptic occupancy prediction, achieving 3.1 RayPQ and 3.9 RayPQ, respectively. These results highlight foundation-model-driven perception as a practical route to training-free 3D scene understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。