提升图像修复模型性能,让变换后图像输出也相应变化。
Enhancing Image Restoration Transformer via Adaptive Translation Equivariance
- 用滑动索引和组件堆叠保持图像平移不变性
- 自适应选择关键值对,兼顾计算效率与感受野
- 适合追求高精度图像修复的开发者使用
平移等变性是图像修复中的基本归纳偏置,确保输入平移时输出也相应平移。现代修复变压器中的注意力机制破坏了这一性质,影响训练收敛与泛化能力。为此,本文提出两种策略:滑动索引(slide indexing)固定操作响应位置,滑动窗口注意力是其典型例子;组件堆叠(component stacking)允许平移等变算子并行或串行排列,构建复杂架构同时保持等变性。但二者仍面临自注意力高开销与滑动窗口固定感受野的矛盾。为此,本文设计自适应滑动索引机制,为每个查询高效选择关键值对,并与全局聚合的关键值对并行拼接。所提网络称为翻译等变自适应变压器(TEAFormer),在多种图像修复任务上评估,结果表明其在有效性、训练收敛性和泛化能力方面均具优势。
原文摘要 · Abstract (English)
Translation equivariance is a fundamental inductive bias in image restoration, ensuring that translated inputs produce translated outputs. Attention mechanisms in modern restoration transformers undermine this property, adversely impacting both training convergence and generalization. To alleviate this issue, we propose two key strategies for incorporating translation equivariance: slide indexing and component stacking. Slide indexing maintains operator responses at fixed positions, with sliding window attention being a notable example, while component stacking enables the arrangement of translation-equivariant operators in parallel or sequentially, thereby building complex architectures while preserving translation equivariance. However, these strategies still create a dilemma in model design between the high computational cost of self-attention and the fixed receptive field associated with sliding window attention. To address this, we develop an adaptive sliding indexing mechanism to efficiently select key-value pairs for each query, which are then concatenated in parallel with globally aggregated key-value pairs. The designed network, called the Translation Equivariance Adaptive Transformer (TEAFormer), is assessed across a variety of image restoration tasks. The results highlight its superiority in terms of effectiveness, training convergence, and generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。