arXiv:2412.18076cs.CVcs.AI2024-12被引 37

解决多模态目标检测中的图像错位问题,提升检测精度与效率。

COMO: Cross-Mamba Interaction and Offset-Guided Fusion for Multimodal Object Detection

  • 采用跨模态Mamba架构实现序列化特征交互,降低计算开销。
  • 利用高层特征缓解相机角度差异导致的位置偏移问题。
  • 设计全局局部扫描机制与偏移引导融合,增强多尺度特征利用。

单模态目标检测在复杂场景下性能易下降,而多模态检测通过融合多源数据可提供更全面的物体特征信息。然而,由于不同传感器采集的图像存在空间错位,直接匹配困难,影响跨模态物体关联。本文提出跨模态Mamba交互与偏移引导融合框架(COMO),利用跨模态Mamba技术构建特征交互方程,实现序列化状态计算,生成交互式融合输出并减少计算负担。同时,基于受错位影响较小的高层特征促进模态间信息传递,缓解因视角与拍摄时间差异带来的位置偏移。此外,跨模态模块引入全局与局部扫描机制,捕捉遥感图像中的局部相关性。为保留低层特征,采用偏移引导融合机制,构建多尺度融合数据立方体,有效提升检测性能。

原文摘要 · Abstract (English)

Single-modal object detection tasks often experience performance degradation when encountering diverse scenarios. In contrast, multimodal object detection tasks can offer more comprehensive information about object features by integrating data from various modalities. Current multimodal object detection methods generally use various fusion techniques, including conventional neural networks and transformer-based models, to implement feature fusion strategies and achieve complementary information. However, since multimodal images are captured by different sensors, there are often misalignments between them, making direct matching challenging. This misalignment hinders the ability to establish strong correlations for the same object across different modalities. In this paper, we propose a novel approach called the CrOss-Mamba interaction and Offset-guided fusion (COMO) framework for multimodal object detection tasks. The COMO framework employs the cross-mamba technique to formulate feature interaction equations, enabling multimodal serialized state computation. This results in interactive fusion outputs while reducing computational overhead and improving efficiency. Additionally, COMO leverages high-level features, which are less affected by misalignment, to facilitate interaction and transfer complementary information between modalities, addressing the positional offset challenges caused by variations in camera angles and capture times. Furthermore, COMO incorporates a global and local scanning mechanism in the cross-mamba module to capture features with local correlation, particularly in remote sensing images. To preserve low-level features, the offset-guided fusion mechanism ensures effective multiscale feature utilization, allowing the construction of a multiscale fusion data cube that enhances detection performance.

多模态检测Mamba目标检测遥感图像

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。