提升视觉Transformer中位置编码效果，让模型更准识别图像位置
Maximizing the Position Embedding for Vision Transformers with Global Average Pooling
- 用分层位置编码增强模型对图像块顺序的感知能力
- 在全局平均池化下保持位置编码与特征的平衡性，提升性能
- 适合改进基于全局平均池化的视觉Transformer结构
在视觉Transformer中，位置编码（PE）对捕捉图像块顺序至关重要。然而，传统结构中将位置编码简单加到令牌嵌入上，限制了其表达能力。已有分层方法通过每层独立传递位置编码并分别归一化，缓解此问题。本文发现，当使用全局平均池化（GAP）而非类别令牌时，该分层结构会出现矛盾结果。为此，提出MPVG方法，在分层结构中最大化位置编码的有效性。研究揭示，位置编码在每层会抵消令牌嵌入值，但原有结构中这种抵消作用不足。通过MPVG强化这一抵消方向性，实验表明该方法在多种视觉任务上均优于现有方法。
原文摘要 · Abstract (English)
In vision transformers, position embedding (PE) plays a crucial role in capturing the order of tokens. However, in vision transformer structures, there is a limitation in the expressiveness of PE due to the structure where position embedding is simply added to the token embedding. A layer-wise method that delivers PE to each layer and applies independent Layer Normalizations for token embedding and PE has been adopted to overcome this limitation. In this paper, we identify the conflicting result that occurs in a layer-wise structure when using the global average pooling (GAP) method instead of the class token. To overcome this problem, we propose MPVG, which maximizes the effectiveness of PE in a layer-wise structure with GAP. Specifically, we identify that PE counterbalances token embedding values at each layer in a layer-wise structure. Furthermore, we recognize that the counterbalancing role of PE is insufficient in the layer-wise structure, and we address this by maximizing the effectiveness of PE through MPVG. Through experiments, we demonstrate that PE performs a counterbalancing role and that maintaining this counterbalancing directionality significantly impacts vision transformers. As a result, the experimental results show that MPVG outperforms existing methods across vision transformers on various tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。