arXiv:2511.06593cs.CV2025-11中稿 · IEEE Transactions …被引 16

提出新型融合框架,提升多模态图像融合效果。

Spatial-Frequency Enhanced Mamba for Multi-Modal Image Fusion

  • 三分支结构同步处理图像融合与重建任务
  • 引入空间频率增强模块,提升特征提取能力
  • 动态融合块实现跨分支自适应特征融合

多模态图像融合(MMIF)旨在整合不同模态的互补信息以生成高信息量图像。现有基于深度学习的方法通常采用卷积神经网络(CNN)或Transformer进行特征提取,但受限于CNN的感受野有限和Transformer计算开销高,性能不理想。近期,Mamba在低复杂度下建模长程依赖方面展现出潜力,适用于MMIF。然而,传统Mamba缺乏完整空间与频率感知,对MMIF不利。同时,图像重建(IR)作为辅助任务已被证明有益,但如何高效利用仍具挑战。为此,本文提出一种名为空间-频率增强Mamba融合(SFMFusion)的新框架。首先设计三分支结构,联合优化MMIF与IR,保留源图像完整内容;其次提出空间-频率增强Mamba块(SFMB),在空间与频率域双重增强Mamba特征表达;最后引入动态融合Mamba块(DFMB),可部署于各分支间实现动态特征融合。大量实验表明,该方法在六个标准数据集上优于多数当前最优方法。代码已开源:https://github.com/SunHui1216/SFMFusion。

原文摘要 · Abstract (English)

Multi-Modal Image Fusion (MMIF) aims to integrate complementary image information from different modalities to produce informative images. Previous deep learning-based MMIF methods generally adopt Convolutional Neural Networks (CNNs) or Transformers for feature extraction. However, these methods deliver unsatisfactory performances due to the limited receptive field of CNNs and the high computational cost of Transformers. Recently, Mamba has demonstrated a powerful potential for modeling long-range dependencies with linear complexity, providing a promising solution to MMIF. Unfortunately, Mamba lacks full spatial and frequency perceptions, which are very important for MMIF. Moreover, employing Image Reconstruction (IR) as an auxiliary task has been proven beneficial for MMIF. However, a primary challenge is how to leverage IR efficiently and effectively. To address the above issues, we propose a novel framework named Spatial-Frequency Enhanced Mamba Fusion (SFMFusion) for MMIF. More specifically, we first propose a three-branch structure to couple MMIF and IR, which can retain complete contents from source images. Then, we propose the Spatial-Frequency Enhanced Mamba Block (SFMB), which can enhance Mamba in both spatial and frequency domains for comprehensive feature extraction. Finally, we propose the Dynamic Fusion Mamba Block (DFMB), which can be deployed across different branches for dynamic feature fusion. Extensive experiments show that our method achieves better results than most state-of-the-art methods on six MMIF datasets. The source code is available at https://github.com/SunHui1216/SFMFusion.

图像融合Mamba多模态特征增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。