融合CNN、Mamba与注意力机制,实现高效通用图像修复。
An Efficient and Mixed Heterogeneous Model for Image Restoration
- 采用三级编码器-解码器结构,分阶段融合CNN、Mamba和注意力模块。
- 在多个图像修复任务中达到领先性能,推理速度优于同类模型。
- 适合需要高效高精度修复的多媒体应用开发者使用。
图像修复(IR)作为基础多媒体处理任务,对下游视觉应用有重要影响。近年来,研究者致力于开发能处理多种退化类型的通用IR模型,以降低模型开发成本与复杂度。现有主流方法基于三种架构范式:卷积神经网络(CNN)、Transformer和Mamba。CNN在高效推理方面表现优异,而Transformer和Mamba擅长捕捉长程依赖与建模全局上下文。尽管各类架构在特定单任务场景中已取得成功,但将异构架构有效融合以共同应对多样化修复挑战的研究仍有限。为此,我们提出RestorMixer,一种基于混合架构融合的高效通用图像修复模型。RestorMixer采用三级编码器-解码器结构,每阶段根据输入分辨率与特征特性定制设计。初始高分辨率阶段使用基于CNN的模块快速提取浅层局部特征;后续阶段融合改进的多方向扫描Mamba模块与多尺度窗口自注意力机制。这种分层自适应设计使模型同时发挥CNN在局部特征提取中的优势、Mamba在全局上下文建模中的能力以及注意力机制在动态特征优化中的作用。大量实验表明,RestorMixer在多个图像修复任务中均达领先性能,且保持高推理效率。官方代码见:https://github.com/ClimBin/RestorMixer。
原文摘要 · Abstract (English)
Image restoration~(IR), as a fundamental multimedia data processing task, has a significant impact on downstream visual applications. In recent years, researchers have focused on developing general-purpose IR models capable of handling diverse degradation types, thereby reducing the cost and complexity of model development. Current mainstream approaches are based on three architectural paradigms: CNNs, Transformers, and Mambas. CNNs excel in efficient inference, whereas Transformers and Mamba excel at capturing long-range dependencies and modeling global contexts. While each architecture has demonstrated success in specialized, single-task settings, limited efforts have been made to effectively integrate heterogeneous architectures to jointly address diverse IR challenges. To bridge this gap, we propose RestorMixer, an efficient and general-purpose IR model based on mixed-architecture fusion. RestorMixer adopts a three-stage encoder-decoder structure, where each stage is tailored to the resolution and feature characteristics of the input. In the initial high-resolution stage, CNN-based blocks are employed to rapidly extract shallow local features. In the subsequent stages, we integrate a refined multi-directional scanning Mamba module with a multi-scale window-based self-attention mechanism. This hierarchical and adaptive design enables the model to leverage the strengths of CNNs in local feature extraction, Mamba in global context modeling, and attention mechanisms in dynamic feature refinement. Extensive experimental results demonstrate that RestorMixer achieves leading performance across multiple IR tasks while maintaining high inference efficiency. The official code can be accessed at https://github.com/ClimBin/RestorMixer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。