提出无需训练的剪枝方法，解决视觉变压器中冗余注意力问题。
ASAP: Attention Sink Anchored Pruning
- 将注意力黑洞建模为概率积累点，通过扩散距离划分关键令牌
- 单次操作实现背景冗余压缩，最高提升48%推理速度
- 适用于图像、视频和多模态任务，无需重新训练
视觉变换器（ViTs）在高分辨率下因自注意力的二次复杂度面临严重计算瓶颈。现有令牌压缩方法依赖局部指标（如单层注意力得分），易受注意力黑洞现象影响——无意义令牌反而被保留，而显著前景对象被忽略。我们提出ASAP（Attention Sink Anchored Pruning），一种无需训练的框架，将该黑洞视为特征。通过将ViT信息流建模为懒惰随机游走，ASAP将黑洞识别为概率质量的主要累积点。基于累积转移矩阵中的扩散距离，采用径向扩散聚类划分令牌，并通过转移权重池化一次性压缩背景冗余。在图像、视频和视觉-语言任务上的大量实验表明，ASAP优于现有最先进方法，最高可提升48%吞吐量，同时保持甚至超越基线准确率。
原文摘要 · Abstract (English)
Vision Transformers (ViTs) face severe computational bottlenecks due to the quadratic complexity of self-attention at high resolutions. Existing token reduction methods rely on local metrics - such as single-layer attention scores - that are inherently vulnerable to the attention sink phenomenon, where uninformative tokens are paradoxically preserved over salient foreground objects. We propose ASAP (Attention Sink Anchored Pruning), a training-free framework that recasts this sink as a feature. Modeling ViT information flow as a Lazy Random Walk, ASAP identifies the sink as a dominant accumulator of probability mass. By computing the diffusion distance to the sink within the cumulative transition matrix, ASAP partitions tokens via Radial Diffusion Clustering and compresses background redundancy through Transition Weight Pooling in a single shot. Extensive experiments across image, video, and vision-language tasks demonstrate ASAP outperforms state-of-the-art methods, accelerating throughput by up to 48% while maintaining - or even exceeding - baseline accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。