融合点云与图像特征,提升自动驾驶3D目标检测精度。
Dual-Domain Homogeneous Fusion with Cross-Modal Mamba and Progressive Decoder for 3D Object Detection
- 在BEV和体素双域进行同质融合,缓解特征压缩问题。
- 跨模态Mamba块解决特征错位,提升场景感知能力。
- 渐进式查询与解码器适合高精度实时检测任务。
在自动驾驶的3D目标检测中,将激光雷达与图像特征融合到统一的鸟瞰图(BEV)域已成为主流方法。然而,该范式受限于过度的特征压缩。尽管部分工作探索密集体素融合以增强特征交互,但面临计算成本高和查询生成困难的问题。此外,两域间的特征错位导致检测精度不理想。为此,本文提出双域同质融合网络(DDHFusion),充分利用BEV与体素域的互补性并克服其缺陷。首先,通过lift-splat-shot及提出的语义感知特征采样(SAFS)模块,将图像特征转换为BEV与稀疏体素表示,后者显著降低计算开销。接着,设计同质体素与BEV融合网络(HVF、HBF),引入新型跨模态Mamba块以解决特征错位,实现全面场景理解。体素特征输出后注入BEV空间,补偿直接高度压缩带来的信息损失。在查询选择阶段,采用贝叶斯域渐进查询生成(PQG)机制,减少因特征压缩导致的漏检。此外,提出渐进式解码器(QD),通过可变形注意力与多模态体素特征混合(MMVFM)块,逐步聚合富含上下文的BEV特征与几何感知的体素特征,实现精准分类与框回归。
原文摘要 · Abstract (English)
Fusing LiDAR and image features in a homogeneous BEV domain has become popular for 3D object detection in autonomous driving. However, this paradigm is constrained by the excessive feature compression. While some works explore dense voxel fusion to enable better feature interaction, they face high computational costs and challenges in query generation. Additionally, feature misalignment in both domains results in suboptimal detection accuracy. To address these limitations, we propose a Dual-Domain Homogeneous Fusion network (DDHFusion), which leverages the complementarily of both BEV and voxel domains while mitigating their drawbacks. Specifically, we first transform image features into BEV and sparse voxel representations using lift-splat-shot and our proposed Semantic-Aware Feature Sampling (SAFS) module. The latter significantly reduces computational overhead by discarding unimportant voxels. Next, we introduce Homogeneous Voxel and BEV Fusion (HVF and HBF) networks for multi-modal fusion within respective domains. They are equipped with novel cross-modal Mamba blocks to resolve feature misalignment and enable comprehensive scene perception. The output voxel features are injected into the BEV space to compensate for the information loss brought by direct height compression. During query selection, the Progressive Query Generation (PQG) mechanism is implemented in the BEV domain to reduce false negatives caused by feature compression. Furthermore, we propose a Progressive Decoder (QD) that sequentially aggregates not only context-rich BEV features but also geometry-aware voxel features with deformable attention and the Multi-Modal Voxel Feature Mixing (MMVFM) block for precise classification and box regression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。