提出DnA注意力机制，提升视觉任务的特征区分度。
DnA: Denoising Attention for Visual Tasks
- 用正负查询分离相关与无关特征，增强注意力判别力。
- 在ImageNet-1K上比基线提升0.8%，视频任务最高增1.8%。
- 适合需要精准注意力的视觉理解场景，如视频分析。
多头注意力（MHA）中的softmax激活函数是视觉感知任务的标准配置。然而，标准softmax会产生噪声注意力模式，削弱关键特征并降低性能。本文提出去噪注意力（DnA）：首先，通过正查询识别属于正确类别的图像特征，负查询识别相关但不相关的特征；随后将这些交互投影到两个具有更大主夹角的子空间中，促进子空间分离，提升判别性。使用ViT-B骨干网络，DnA在ImageNet-1K上相比基线实现0.8%的绝对提升。此外，在视频理解任务中，视频变换器（Video Transformers）和视频大语言模型（Video LLMs）分别获得1.8%和0.5%的改进。广泛的实证分析验证了双子空间设计及DnA的去噪效果。
原文摘要 · Abstract (English)
The softmax activation in multihead attention (MHA) is the de facto standard for attention-based models in visual perception tasks. However, standard softmax can produce noisy attention patterns that dilute relevant features and degrade its performance. In this paper, we propose Denoising Attention or DnA, in which, first, a positive query identifies which image features belong to the correct class, and a negative query identifies closely associated but irrelevant image features. DnA then projects these interactions into two distinct subspaces with larger principal angles, promoting subspace separation and improved discriminability. Using a ViT-B backbone, our proposed DnA achieves an absolute gain of 0.8% on ImageNet-1K compared to the baseline. We further show improvements across multiple visual understanding tasks, including video understanding with video transformers (1.8%) and video LLMs (0.5%). Our extensive empirical analyses justify the design choices involving two interacting subspaces and the denoising effect of DnA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。