通过频率感知策略减少视觉Transformer的冗余令牌，提升效率与精度。
Frequency-Aware Token Reduction for Efficient Vision Transformer
- 将令牌分为高频与低频，保留高频，聚合低频为直流项。
- 在保持性能的同时，计算量显著降低，且缓解秩坍缩与过平滑现象。
- 适用于追求高效视觉Transformer推理的开发者与研究者。
视觉Transformer在各类计算机视觉任务中表现出色，但其关于令牌长度的二次计算复杂度仍是重大挑战。为应对这一问题，已有多种令牌压缩方法被广泛研究。然而，现有方法常忽视自注意力机制中的频率特性，如秩坍缩和过平滑现象。本文提出一种频率感知的令牌压缩策略，在提升计算效率的同时保持性能，有效缓解秩坍缩问题。该方法将令牌分为高频与低频两类：高频令牌被选择性保留，而低频令牌则聚合为一个紧凑的直流令牌以保留关键低频成分。通过大量实验与分析，我们证明该方法在显著降低计算开销的同时，显著提升准确率，并有效缓解秩坍缩与过平滑现象。此外，我们对先前方法进行了分析，揭示了其隐含的频率特性及局限性。
原文摘要 · Abstract (English)
Vision Transformers have demonstrated exceptional performance across various computer vision tasks, yet their quadratic computational complexity concerning token length remains a significant challenge. To address this, token reduction methods have been widely explored. However, existing approaches often overlook the frequency characteristics of self-attention, such as rank collapsing and over-smoothing phenomenon. In this paper, we propose a frequency-aware token reduction strategy that improves computational efficiency while preserving performance by mitigating rank collapsing. Our method partitions tokens into high-frequency tokens and low-frequency tokens. high-frequency tokens are selectively preserved, while low-frequency tokens are aggregated into a compact direct current token to retain essential low-frequency components. Through extensive experiments and analysis, we demonstrate that our approach significantly improves accuracy while reducing computational overhead and mitigating rank collapsing and over smoothing. Furthermore, we analyze the previous methods, shedding light on their implicit frequency characteristics and limitations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。