提出交互式空间-频率融合框架，提升多模态图像融合效果
Interactive Spatial-Frequency Fusion Mamba for Multi-Modal Image Fusion
- 设计交互式空间-频率融合机制，跨模态引导特征增强
- 在6个数据集上超越现有方法，显著保留纹理与关键信息
- 适合图像融合、医学影像处理等需要细节保持的场景
多模态图像融合（MMIF）旨在结合不同模态的图像，生成保留纹理细节并完整保存重要信息的融合图像。近期方法引入频域信息以增强空间特征，但大多采用简单的串行或并行融合，缺乏交互。本文提出一种新型交互式空间-频率融合Mamba（ISFM）框架。首先通过模态特定提取器（MSE）提取各模态特征，利用线性计算复杂度建模图像长程依赖。为有效利用频域信息，提出多尺度频域融合（MFF），自适应融合多尺度下的低频与高频成分，实现稳健的频域表征。更重要的是，提出交互式空间-频率融合（ISF），让频域特征跨模态引导空间特征，增强互补表示。在六个MMIF数据集上进行大量实验，结果表明，ISFM优于当前主流方法。代码已开源。
原文摘要 · Abstract (English)
Multi-Modal Image Fusion (MMIF) aims to combine images from different modalities to produce fused images, retaining texture details and preserving significant information. Recently, some MMIF methods incorporate frequency domain information to enhance spatial features. However, these methods typically rely on simple serial or parallel spatial-frequency fusion without interaction. In this paper, we propose a novel Interactive Spatial-Frequency Fusion Mamba (ISFM) framework for MMIF. Specifically, we begin with a Modality-Specific Extractor (MSE) to extract features from different modalities. It models long-range dependencies across the image with linear computational complexity. To effectively leverage frequency information, we then propose a Multi-scale Frequency Fusion (MFF). It adaptively integrates low-frequency and high-frequency components across multiple scales, enabling robust representations of frequency features. More importantly, we further propose an Interactive Spatial-Frequency Fusion (ISF). It incorporates frequency features to guide spatial features across modalities, enhancing complementary representations. Extensive experiments are conducted on six MMIF datasets. The experimental results demonstrate that our ISFM can achieve better performances than other state-of-the-art methods. The source code is available at https://github.com/Namn23/ISFM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。