融合4D雷达与摄像头,实现强鲁棒的3D占位预测。
MetaOcc: Spatio-Temporal Fusion of Surround-View 4D Radar and Camera for 3D Occupancy Prediction with Dual Training Strategies
- 设计雷达高度自注意力模块,提升垂直空间感知能力。
- 提出分层多尺度跨模态融合,缓解时空错位问题。
- 半监督训练仅用50%标注数据达90%性能,降低标注成本。
可靠的3D占位预测对自动驾驶至关重要,尤其在恶劣天气下视觉系统表现受限时。本文提出MetaOcc,一种基于环绕式4D雷达与摄像头的全景3D占位预测框架。为解决将激光雷达编码器直接用于稀疏雷达数据的局限性,提出雷达高度自注意力模块,增强垂直空间推理能力。进一步设计分层多尺度跨模态融合策略,实现模态间与时间维度上的自适应局部-全局融合,缓解时空错位并丰富特征表示。为减少对昂贵点云标注的依赖,提出基于开集分割器的伪标签生成流程,实现半监督训练:仅使用50%真实标签即可达到全监督性能的90%。大量实验表明,MetaOcc在全监督下于OmniHD-Scenes数据集上实现+0.47 SC IoU和+4.02 mIoU,于SurroundOcc-nuScenes上实现+1.16 SC IoU和+1.24 mIoU,优于此前方法,展现出跨传感器域与训练条件的可扩展性与鲁棒性。代码与数据见https://github.com/LucasYang567/MetaOcc。
原文摘要 · Abstract (English)
Robust 3D occupancy prediction is essential for autonomous driving, particularly under adverse weather conditions where traditional vision-only systems struggle. While the fusion of surround-view 4D radar and cameras offers a promising low-cost solution, effectively extracting and integrating features from these heterogeneous sensors remains challenging. This paper introduces MetaOcc, a novel multi-modal framework for omnidirectional 3D occupancy prediction that leverages both multi-view 4D radar and images. To address the limitations of directly applying LiDAR-oriented encoders to sparse radar data, we propose a Radar Height Self-Attention module that enhances vertical spatial reasoning and feature extraction. Additionally, a Hierarchical Multi-scale Multi-modal Fusion strategy is developed to perform adaptive local-global fusion across modalities and time, mitigating spatio-temporal misalignments and enriching fused feature representations. To reduce reliance on expensive point cloud annotations, we further propose a pseudo-label generation pipeline based on an open-set segmentor. This enables a semi-supervised strategy that achieves 90% of the fully supervised performance using only 50% of the ground truth labels, offering an effective trade-off between annotation cost and accuracy. Extensive experiments demonstrate that MetaOcc under full supervision achieves state-of-the-art performance, outperforming previous methods by +0.47 SC IoU and +4.02 mIoU on the OmniHD-Scenes dataset, and by +1.16 SC IoU and +1.24 mIoU on the SurroundOcc-nuScenes dataset. These results demonstrate the scalability and robustness of MetaOcc across sensor domains and training conditions, paving the way for practical deployment in real-world autonomous systems. Code and data are available at https://github.com/LucasYang567/MetaOcc.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。