融合Mamba与Transformer,提升图像修复效率与效果
MatIR: A Hybrid Mamba-Transformer Image Restoration Model
- 交替使用Mamba与Transformer模块提取特征
- 在多个数据集上优于现有方法,峰值信噪比提升0.1~0.3dB
- 适合追求高效高质图像修复的开发者
近年来,基于Transformer的模型在图像修复领域取得显著进展,因其能捕捉复杂的上下文特征。近期,Mamba模型凭借处理长程依赖和更高的计算效率在计算机视觉中崭露头角,但在上下文学习能力上仍落后于Transformer。为克服两者局限,我们提出一种名为MatIR的Mamba-Transformer混合图像修复模型。具体地,MatIR通过交叉循环Transformer层与Mamba层的块来提取特征,充分发挥两者的优点。在Mamba模块中,引入图像修补状态空间(IRSS)模块,沿四条扫描路径处理长序列数据,实现高效计算。在Transformer模块中,结合三角窗局部注意力与通道全局注意力,有效激活更广泛像素范围内的注意力机制。大量实验与消融研究验证了该方法的有效性。
原文摘要 · Abstract (English)
In recent years, Transformers-based models have made significant progress in the field of image restoration by leveraging their inherent ability to capture complex contextual features. Recently, Mamba models have made a splash in the field of computer vision due to their ability to handle long-range dependencies and their significant computational efficiency compared to Transformers. However, Mamba currently lags behind Transformers in contextual learning capabilities. To overcome the limitations of these two models, we propose a Mamba-Transformer hybrid image restoration model called MatIR. Specifically, MatIR cross-cycles the blocks of the Transformer layer and the Mamba layer to extract features, thereby taking full advantage of the advantages of the two architectures. In the Mamba module, we introduce the Image Inpainting State Space (IRSS) module, which traverses along four scan paths to achieve efficient processing of long sequence data. In the Transformer module, we combine triangular window-based local attention with channel-based global attention to effectively activate the attention mechanism over a wider range of image pixels. Extensive experimental results and ablation studies demonstrate the effectiveness of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。