发现视觉Transformer中两类关键冗余令牌，提出高效近似方法
Artifacts and Attention Sinks: Structured Approximations for Efficient Vision Transformers
- 识别出激活值异常高的'注意力陷阱'与'伪影令牌'及其相互抑制机制
- 提出FNA方法，实现线性复杂度自注意力计算，推理速度提升显著
- 无需训练的掩码策略有效降噪，适合部署在资源受限场景
视觉Transformer虽广泛应用，但其内部机理仍不清晰。本文分析发现，存在激活范数极高的'注意力陷阱'令牌和推断过程中产生的'伪影令牌'，二者通过注意力机制相互抑制，调控网络信息流动。基于此，提出无需训练的Fast Nyström Attention（FNA）方法，利用这些令牌的结构化模式，在线性时间和空间复杂度下近似自注意力。同时提出掩码策略抑制噪声，带来微小性能提升且开销可忽略。在多个主流预训练视觉骨干网络上验证，于图像检索、分类、分割及视觉问答任务中均表现良好，同时显著降低计算开销。
原文摘要 · Abstract (English)
Vision transformers have emerged as a powerful tool across a wide range of applications, yet their inner workings remain only partially understood. In this work, we examine the phenomenon of massive tokens - tokens with exceptionally high activation norms that act as attention sinks - and artifact tokens that emerge as a byproduct during inference. Our analysis reveals that these tokens mutually suppress one another through the attention mechanism, playing a critical role in regulating information flow within the network. Leveraging these insights, we introduce Fast Nyström Attention (FNA), a training-free method that approximates self-attention in linear time and space by exploiting the structured patterns formed by massive and artifact tokens. Additionally, we propose a masking strategy to mitigate noise from these tokens, yielding modest performance gains at virtually no cost. We evaluate our approach on popular pretrained vision backbones and demonstrate competitive performance on retrieval, classification, segmentation, and visual question answering (VQA), all while reducing computational overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。