arXiv:2507.14643cs.CV2025-07中稿 · Information Fusion被引 14

提出MS2Fusion框架,用状态空间模型融合多光谱特征,提升检测性能与泛化能力。

Multispectral State-Space Feature Fusion: Bridging Shared and Cross-Parametric Interactions for Object Detection

  • 双路径机制:跨参数分支挖掘互补信息,共享参数分支对齐语义结构
  • 在FLIR、M3FD等数据集上显著超越现有方法,提升平均精度达数个百分点
  • 框架通用性强,可直接用于红外-可见光分割与显著目标检测任务

当前多光谱目标检测的特征融合面临两大瓶颈:过度偏好局部互补特征而忽视跨模态共享语义,影响泛化能力;感受野大小与计算复杂度之间的权衡制约了可扩展建模。为此,本文提出基于状态空间模型(SSM)的多光谱状态空间特征融合框架MS2Fusion,通过双路径参数交互机制实现高效精准融合。第一路跨参数分支利用跨注意力机制和SSM中的跨模态隐状态解码,挖掘异源特征的互补信息;第二路共享参数分支通过联合嵌入与参数共享,实现跨模态语义对齐与结构一致。两路在统一框架中联合优化,兼顾功能互补与共享语义空间。在主流基准如FLIR、M3FD和LLVIP上的大量实验表明,MS2Fusion显著优于现有先进方法。此外,该框架具有强泛化性,无需专门设计即在RGB-T语义分割与RGBT显著目标检测任务中达到领先水平。代码将开源于https://github.com/61s61min/MS2Fusion.git。

原文摘要 · Abstract (English)

Modern multispectral feature fusion for object detection faces two critical limitations: (1) Excessive preference for local complementary features over cross-modal shared semantics adversely affects generalization performance; and (2) The trade-off between the receptive field size and computational complexity present critical bottlenecks for scalable feature modeling. Addressing these issues, a novel Multispectral State-Space Feature Fusion framework, dubbed MS2Fusion, is proposed based on the state space model (SSM), achieving efficient and effective fusion through a dual-path parametric interaction mechanism. More specifically, the first cross-parameter interaction branch inherits the advantage of cross-attention in mining complementary information with cross-modal hidden state decoding in SSM. The second shared-parameter branch explores cross-modal alignment with joint embedding to obtain cross-modal similar semantic features and structures through parameter sharing in SSM. Finally, these two paths are jointly optimized with SSM for fusing multispectral features in a unified framework, allowing our MS2Fusion to enjoy both functional complementarity and shared semantic space. In our extensive experiments on mainstream benchmarks including FLIR, M3FD and LLVIP, our MS2Fusion significantly outperforms other state-of-the-art multispectral object detection methods, evidencing its superiority. Moreover, MS2Fusion is general and applicable to other multispectral perception tasks. We show that, even without specific design, MS2Fusion achieves state-of-the-art results on RGB-T semantic segmentation and RGBT salient object detection, showing its generality. The source code will be available at https://github.com/61s61min/MS2Fusion.git.

多光谱目标检测状态空间模型特征融合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。