提升图像修复与生成效率,突破高分辨率图像处理瓶颈
Fast and Accurate Image Restoration and Generation with Rank Enhanced Linear Attention
- 引入轻量级深度卷积增强线性注意力的特征表达能力
- 在7个图像修复任务中超越现有方法,计算开销显著降低
- 适用于高分辨率图像处理,适合追求效率的视觉生成研究
基于Transformer的模型在图像修复(IR)任务中表现优异,但自注意力机制的二次复杂度限制了其在高分辨率图像上的应用。现有方法通过稀疏或窗口化注意力缓解此问题,但削弱了全局上下文建模能力。线性注意力作为软最大注意力的变体,在保持线性复杂度的同时具备良好的全局建模潜力。然而,原始线性注意力在图像修复任务中性能下降明显,主要源于其注意力图的低秩特性。为此,本文提出秩增强线性注意力(RELA),通过引入轻量级深度卷积丰富特征表示。基于RELA,我们构建了高效视觉变压器LAformer,摒弃了软最大和窗口移位等硬件低效操作,支持高分辨率图像的高效处理。在7项图像修复任务及21个基准上进行的大量实验表明,LAformer优于当前最优方法,并具有显著的计算优势。此外,我们将LAformer扩展至基于扩散和流的视觉生成任务,展现出其作为DiT和SiT的有力替代方案的潜力。代码与模型已开源。
原文摘要 · Abstract (English)
Transformer-based models have made remarkable progress in image restoration (IR) tasks. However, the quadratic complexity of self-attention in Transformer hinders its applicability to high-resolution images. Existing methods mitigate this issue with sparse or window-based attention, yet inherently limit global context modeling. Linear attention, a variant of softmax attention, demonstrates promise in global context modeling while maintaining linear complexity, offering a potential solution to the above challenge. Despite its efficiency benefits, vanilla linear attention suffers from a significant performance drop in IR, largely due to the low-rank nature of its attention map. To counter this, we propose Rank Enhanced Linear Attention (RELA), a simple yet effective method that enriches feature representations by integrating a lightweight depthwise convolution. Building upon RELA, we propose an efficient and effective Vision Transformer, named LAformer. LAformer eliminates hardware-inefficient operations such as softmax and window shifting, enabling efficient processing of high-resolution images. Extensive experiments across 7 IR tasks and 21 benchmarks demonstrate that LAformer outperforms SOTA methods and offers significant computational advantages. Furthermore, we extend LAformer to diffusion-based and flow-based visual generation, showcasing its strong potential as a competitive alternative to DiT and SiT. Code and models are available at https://github.com/shallowdream204/LAformer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。