改进注意力机制，同时利用垂直与投影分量提升视觉任务表现
Improved Belief-Attention in Vision Task
- 新模块Belief2-Attention融合垂直与投影分量，增强信息表达
- 引入额外内积矩阵ZZ^T，捕捉更丰富的标记相关性，提升性能
- 在图像分类与分割任务中验证有效，适用于追求精度的视觉模型
近期提出的Belief-Attention通过将软注意力加权求和结果对原始值向量做正交投影，并取垂直分量作为残差信号以提升Transformer性能。本文首先进行消融实验，发现投影分量同样携带标记相关性信息，不应被忽略。为此，我们提出扩展Belief-Attention，同时利用垂直与投影分量：投影分量经激活函数处理并线性映射后，与当前标记合并。该结构可视为新注意力块内的两层前馈网络。此外，标准注意力通过内积矩阵QK^T捕捉标记相关性，我们进一步引入额外内积矩阵ZZ^T增强表达能力。新模块称为Belief2-Attention，理论证明其表达能力强于标准注意力。我们在图像分类与分割任务中验证了其有效性。
原文摘要 · Abstract (English)
Recently, Belief-Attention \cite{Guoqiang25BeliefAttention} has been proposed by first performing an orthogonal projection of the softmax-based weighted summation of $V$ vectors with respect to the original $V$ vectors and then taking the perpendicular component as the residual signal in Transformer for performance improvement. In this paper, we first conduct an ablation study showing the projected component also carries information about the token correlation, which should not be ignored. We then propose to extend Belief-Attention by making use of both the perpendicular and projected components. In particular, the projected component goes through certain activation function and then a linear mapping before merging with the considered token. Conceptually speaking, the neural block for the projected component can be viewed as a two-layer feedforward network (FFN) within the new attention block. It is also noted that standard attention captures the token correlation via the inner-product matrix $QK^T$. We propose to introduce an additional inner-product matrix $ZZ^T$ to $QK^T$ to capture richer token correlation. We refer to the new module as Belief2-Attention. It can be easily shown that Belief2-Attention is more expressive than standard Attention. We then verify the effectiveness of Belief2-Attention for vision tasks of image classification and segmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。