通过分析注意力图冗余，实现视觉Transformer的极致量化。
Exploiting Information Redundancy in Attention Maps for Extreme Quantization of Vision Transformers
- 用香农熵衡量注意力头信息量，识别冗余头。
- 低熵注意力图冻结并低精度量化，减少计算。
- 在ImageNet-1k上实现≤20%稀疏度下精度不降反升。
Transformer模型依赖多头自注意力（MHSA）机制，但其高计算复杂度和内存开销限制了边缘部署。本文分析注意力图中的信息冗余，通过香农熵量化各注意力头的信息量，发现熵较低（行为更确定）的头贡献较少信息，从而提出针对性压缩策略。基于此，我们提出熵注意力图（EAM）模型：冻结低熵注意力图权重，并将其量化至低精度以避免冗余重算。在ImageNet-1k上的实证表明，EAM在注意力图稀疏度≤20%时达到相似或更高准确率，对DeiT和Swin Transformer模型均表现优异，且在更高稀疏度下仍具竞争力。
原文摘要 · Abstract (English)
Transformer models rely on Multi-Head Self-Attention (MHSA) mechanisms, where each attention head contributes to the final representation. However, their computational complexity and high memory demands due to MHSA hinders their deployment at the edge. In this work, we analyze and exploit information redundancy in attention maps to accelerate model inference. By quantifying the information captured by each attention head using Shannon entropy, our analysis reveals that attention heads with lower entropy, i.e., exhibiting more deterministic behavior, tend to contribute less information, motivating targeted compression strategies. Relying on these insights, we propose Entropy Attention Maps (EAM), a model that freezes the weights of low-entropy attention maps and quantizes these values to low precision to avoid redundant re-computation. Empirical validation on ImageNet-1k shows that EAM achieves similar or higher accuracy at $\leq$20\% sparsity in attention maps and competitive performance beyond this level for the DeiT and Swin Transformer models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。