用雷达相机联合增强目标实例感知,提升3D检测精度。
Boosting Instance Awareness via Cross-View Correlation with 4D Radar and Camera for 3D Object Detection

- 通过分割与深度引导定位,抑制视角转换中的背景噪声。
- 引入跨视图激活机制,将2D实例特征注入鸟瞰图空间。
- 在多个数据集上达到顶尖性能,适合自动驾驶感知系统开发。
4D毫米波雷达因其鲁棒性和低成本,成为自动驾驶中极具前景的传感模态。然而,其稀疏且弱化的几何线索导致可靠的目标实例激活困难,限制了现有雷达-相机融合方法的效果。基于鸟瞰图(BEV)的融合虽能提供全局场景理解,但实例关注不足;基于透视图的融合虽可捕捉实例细节,却缺乏整体上下文。为此,我们提出SIFormer——一种用于4D雷达与相机融合的场景-实例感知变换器。SIFormer首先通过分割与深度引导定位,在视图变换过程中抑制背景噪声;随后引入跨视图激活机制,将2D实例线索注入鸟瞰图空间,实现在弱雷达几何条件下的可靠实例感知;最后,采用基于Transformer的融合模块,聚合互补的图像语义与雷达几何信息,实现鲁棒感知。实验表明,该方法在View-of-Delft、TJ4DRadSet和NuScenes数据集上均达到当前最优性能。源代码已开源:github.com/shawnnnkb/SIFormer。
原文摘要 · Abstract (English)
4D millimeter-wave radar has emerged as a promising sensing modality for autonomous driving due to its robustness and affordability. However, its sparse and weak geometric cues make reliable instance activation difficult, limiting the effectiveness of existing radar-camera fusion paradigms. BEV-level fusion offers global scene understanding but suffers from weak instance focus, while perspective-level fusion captures instance details but lacks holistic context. To address these limitations, we propose SIFormer, a scene-instance aware transformer for 3D object detection using 4D radar and camera. SIFormer first suppresses background noise during view transformation through segmentation- and depth-guided localization. It then introduces a cross-view activation mechanism that injects 2D instance cues into BEV space, enabling reliable instance awareness under weak radar geometry. Finally, a transformer-based fusion module aggregates complementary image semantics and radar geometry for robust perception. As a result, with the aim of enhancing instance awareness, SIFormer bridges the gap between the two paradigms, combining their complementary strengths to address inherent sparse nature of radar and improve detection accuracy. Experiments demonstrate that SIFormer achieves state-of-the-art performance on View-of-Delft, TJ4DRadSet and NuScenes datasets. Source code is available at github.com/shawnnnkb/SIFormer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。