用线性Transformer与Mamba融合，实现高效多模态图像融合
Why mamba is effective? Exploit Linear Transformer-Mamba Network for Multi-Modality Image Fusion
- 双分支结构分别提取通道与位置信息，通过交互模块融合
- 在红外-可见光和医学图像融合上均超越现有方法
- 适合需要高效跨模态特征融合的视觉任务
多模态图像融合旨在整合不同来源图像的优势，生成高质量融合图像。然而，现有特征提取与融合方法或受局部缩减偏差和推理时静态参数限制（如CNN），或受限于二次计算复杂度（如Transformer），难以有效提取与融合特征。为此，我们提出一种名为Tmamba的双分支图像融合网络，结合线性Transformer与Mamba，兼具全局建模能力且保持线性复杂度。由于Transformer与Mamba结构差异，两分支分别提取通道与位置信息。设计了T-M交互结构，利用全局可学习参数与卷积层分别传递位置与通道信息。进一步在注意力层面提出跨模态交互，以获得跨模态注意力。实验表明，Tmamba在多个融合任务中表现优异，包括红外-可见光图像融合与医学图像融合。代码与模型检查点将在同行评审后公开。
原文摘要 · Abstract (English)
Multi-modality image fusion aims to integrate the merits of images from different sources and render high-quality fusion images. However, existing feature extraction and fusion methods are either constrained by inherent local reduction bias and static parameters during inference (CNN) or limited by quadratic computational complexity (Transformers), and cannot effectively extract and fuse features. To solve this problem, we propose a dual-branch image fusion network called Tmamba. It consists of linear Transformer and Mamba, which has global modeling capabilities while maintaining linear complexity. Due to the difference between the Transformer and Mamba structures, the features extracted by the two branches carry channel and position information respectively. T-M interaction structure is designed between the two branches, using global learnable parameters and convolutional layers to transfer position and channel information respectively. We further propose cross-modal interaction at the attention level to obtain cross-modal attention. Experiments show that our Tmamba achieves promising results in multiple fusion tasks, including infrared-visible image fusion and medical image fusion. Code with checkpoints will be available after the peer-review process.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。