融合语义与深度信息,提升3D占位预测精度与实时性。
SDGOCC: Semantic and Depth-Guided Bird's-Eye View Transformation for 3D Multimodal Occupancy Prediction
- 通过语义与深度联合引导的视角转换,精准构建深度分布。
- 在Occ3D-nuScenes上达到实时性能,SurroundOcc上表现优异。
- 适合自动驾驶中需要高精度占位感知的研究与工程应用。
多模态3D占位预测在自动驾驶领域备受关注。然而,现有方法多为单模态:基于相机的方法缺乏深度信息,而基于激光雷达的方法易受遮挡影响。当前轻量级方法主要依赖Lift-Splat-Shoot(LSS)流程,存在深度估计不准且未能充分挖掘3D激光雷达点的几何与语义信息的问题。为此,我们提出新型多模态占位预测网络SDG-OCC,结合联合语义与深度引导的视角转换,以及融合到占位驱动的主动蒸馏机制。增强的视角转换通过扩散与双线性离散化整合像素语义与对应点深度,构建精确深度分布;融合到占位驱动的主动蒸馏则从多模态数据中提取丰富语义,并基于激光雷达识别区域选择性地向图像特征传递知识。最后,为优化性能,引入仅使用融合的SDG-Fusion和融合加蒸馏的SDG-KL,实现更快推理。本方法在Occ3D-nuScenes数据集上实现最先进(SOTA)性能并支持实时处理,在更具挑战性的SurroundOcc-nuScenes数据集上表现相当,证明了其有效性与鲁棒性。代码将发布于https://github.com/DzpLab/SDGOCC。
原文摘要 · Abstract (English)
Multimodal 3D occupancy prediction has garnered significant attention for its potential in autonomous driving. However, most existing approaches are single-modality: camera-based methods lack depth information, while LiDAR-based methods struggle with occlusions. Current lightweight methods primarily rely on the Lift-Splat-Shoot (LSS) pipeline, which suffers from inaccurate depth estimation and fails to fully exploit the geometric and semantic information of 3D LiDAR points. Therefore, we propose a novel multimodal occupancy prediction network called SDG-OCC, which incorporates a joint semantic and depth-guided view transformation coupled with a fusion-to-occupancy-driven active distillation. The enhanced view transformation constructs accurate depth distributions by integrating pixel semantics and co-point depth through diffusion and bilinear discretization. The fusion-to-occupancy-driven active distillation extracts rich semantic information from multimodal data and selectively transfers knowledge to image features based on LiDAR-identified regions. Finally, for optimal performance, we introduce SDG-Fusion, which uses fusion alone, and SDG-KL, which integrates both fusion and distillation for faster inference. Our method achieves state-of-the-art (SOTA) performance with real-time processing on the Occ3D-nuScenes dataset and shows comparable performance on the more challenging SurroundOcc-nuScenes dataset, demonstrating its effectiveness and robustness. The code will be released at https://github.com/DzpLab/SDGOCC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。