融合Transformer与Mamba,提升雨滴去除的长程依赖建模能力。
A Hybrid Transformer-Mamba Network for Single Image Deraining
- 双分支结构:频域Transformer捕捉长程依赖,Mamba处理局部与全局信息。
- 提出频谱相干损失,更好还原清晰图像的内在信号关系。
- 在真实雨天图像上表现优于现有方法,适合实际场景应用。
现有去雨Transformer多采用固定窗口或通道维度的自注意力机制,难以充分挖掘非局部感受野。针对此问题,本文提出一种新型双分支混合Transformer-Mamba网络TransMamba,旨在有效捕捉长距离雨滴相关依赖。基于雨渍与背景在频域特征上的差异先验,第一分支设计频带化Transformer模块,在频域通道维度上执行自注意力,增强长程依赖建模能力;同时引入频谱增强前馈模块,聚合频域特征。第二分支采用级联双向状态空间模型的Mamba层,进一步捕捉局部与全局信息。编码器与解码器各阶段均进行通道维拼接与通道压缩融合,实现多尺度信息高效整合。为更好地重建干净图像中的内在信号级关系,还设计了频谱相干损失函数。在多个数据集及真实雨天图像上的大量实验表明,本方法显著优于当前最优技术。
原文摘要 · Abstract (English)
Existing deraining Transformers employ self-attention mechanisms with fixed-range windows or along channel dimensions, limiting the exploitation of non-local receptive fields. In response to this issue, we introduce a novel dual-branch hybrid Transformer-Mamba network, denoted as TransMamba, aimed at effectively capturing long-range rain-related dependencies. Based on the prior of distinct spectral-domain features of rain degradation and background, we design a spectral-banded Transformer blocks on the first branch. Self-attention is executed within the combination of the spectral-domain channel dimension to improve the ability of modeling long-range dependencies. To enhance frequency-specific information, we present a spectral enhanced feed-forward module that aggregates features in the spectral domain. In the second branch, Mamba layers are equipped with cascaded bidirectional state space model modules to additionally capture the modeling of both local and global information. At each stage of both the encoder and decoder, we perform channel-wise concatenation of dual-branch features and achieve feature fusion through channel reduction, enabling more effective integration of the multi-scale information from the Transformer and Mamba branches. To better reconstruct innate signal-level relations within clean images, we also develop a spectral coherence loss. Extensive experiments on diverse datasets and real-world images demonstrate the superiority of our method compared against the state-of-the-art approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。