将Swin与Retentive网络结合,用门控机制提升视觉注意力效果。
Gated-SwinRMT: Unifying Swin Windowed Attention with Retentive Manhattan Decay via Input-Dependent Gating
- 通过输入相关门控,融合Swin的窗口注意力与RMT的曼哈顿衰减机制。
- 在Mini-ImageNet上达80.22%准确率,比基线高6.48个百分点。
- 适合关注高效视觉建模与局部注意力设计的研究者。
我们提出Gated-SwinRMT,一种融合Swin Transformer分块窗口注意力与Retentive Networks(RMT)曼哈顿距离空间衰减特性的混合视觉变换器,通过输入依赖门控增强性能。自注意力被分解为每个移位窗口内的逐宽与逐高保留传递,每头使用指数衰减掩码实现二维局部先验,无需学习的位置偏置。提出两种变体:Gated-SwinRMT-SWAT 将softmax替换为sigmoid,采用平衡的ALiBi斜率并乘性后激活衰减,值投影通过SwiGLU实现,归一化输出隐式抑制无效注意力得分;Gated-SwinRMT-Retention 保留softmax归一化保留,引入加性对数空间衰减偏置,并加入显式G1 sigmoid门控——从块输入投影而来,在局部上下文增强(LCE)后、输出投影前应用,以缓解低秩$W_V \cdot W_O$瓶颈并实现输入依赖的输出抑制。在相同训练协议下评估两个变体,仅使用单张GPU资源,分别在Mini-ImageNet(224×224,100类)和CIFAR-10(32×32,10类)上测试。参数量约77–79M时,Gated-SwinRMT-SWAT在Mini-ImageNet上达到80.22%的顶1准确率,而Gated-SwinRMT-Retention为78.20%,均高于6.48个百分点的RMT基线。在小尺寸特征图的CIFAR-10上,优势从+6.48个百分点压缩至+0.56个百分点。
原文摘要 · Abstract (English)
We introduce Gated-SwinRMT, a family of hybrid vision transformers that combine the shifted-window attention of the Swin Transformer with the Manhattan-distance spatial decay of Retentive Networks (RMT), augmented by input-dependent gating. Self-attention is decomposed into consecutive width-wise and height-wise retention passes within each shifted window, where per-head exponential decay masks provide a two-dimensional locality prior without learned positional biases. Two variants are proposed.Gated-SwinRMT-SWAT substitutes softmax with sigmoid activation, implements balanced ALiBi slopes with multiplicative post-activation spatial decay, and gates the value projection via SwiGLU; the Normalized output implicitly suppresses uninformative attention scores. \textbf{Gated-SwinRMT-Retention} retains softmax-normalized retention with an additive log-space decay bias and incorporates an explicit G1 sigmoid gate -- projected from the block input and applied after local context enhancement (LCE) but prior to the output projection~$W_O$ -- to alleviate the low-rank $W_V \!\cdot\! W_O$ bottleneck and enable input-dependent suppression of attended outputs. We assess both variants on Mini-ImageNet ($224{\times}224$, 100 classes) and CIFAR-10 ($32{\times}32$, 10 classes) under identical training protocols, utilizing a single GPU due to resource limitations. At ${\approx}77$--$79$\,M parameters, Gated-SwinRMT-SWAT achieves $80.22\%$ and Gated-SwinRMT-Retention $78.20\%$ top-1 test accuracy on Mini-ImageNet, compared with $73.74\%$ for the RMT baseline. On CIFAR-10 -- where small feature maps cause the adaptive windowing mechanism to collapse attention to global scope -- the accuracy advantage compresses from $+6.48$\,pp to $+0.56$\,pp.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。