用瑞尼熵识别关键视觉令牌，提升ViT早期层剪枝准确性
Rényi Entropy: A New Token Pruning Metric for Vision Transformers
- 基于瑞尼熵设计无需训练的令牌重要性度量方法
- 在多类视觉模型上实现比现有方法更优的剪枝效果
- 特别适合需要高效推理的早期层剪枝场景
视觉变换器（ViTs）虽达到顶尖性能，但自注意力机制存在O(N²)复杂度，高分辨率输入下推理成本高昂。为缓解此瓶颈，令牌剪枝成为关键加速手段。现有方法多依赖[CLS]令牌评估区块重要性，但在网络早期层语义表征尚未成熟时，[CLS]令牌不可靠，导致剪枝误判与信息损失。本文提出一种无需训练的令牌重要性度量方法Col-Ln，基于瑞尼熵从网络第一层即可识别有效令牌，实现更可靠的剪枝。在多种ViT及大视觉语言模型（LVLMs）上的实验表明，该方法在多个基准测试中持续优于当前最优剪枝方法。
原文摘要 · Abstract (English)
Vision Transformers (ViTs) achieve state-of-the-art performance but suffer from the $O(N^2)$ complexity of self-attention, making inference costly for high-resolution inputs. To address this bottleneck, token pruning has emerged as a critical technique to accelerate inference. Most existing methods rely on the [CLS] token to estimate patch importance. However, we argue that the [CLS] token can be unreliable in early layers where semantic representations are still immature. As a result, pruning in the early layer often leads to inaccurate importance estimation and unnecessary information loss. In this work, we propose a training-free token importance metric, namely Col-Ln, which is derived from Rényi entropy that enables the identification of informative tokens from the first layer of the network, thereby enabling more reliable pruning in token reduction. Extensive experiments on ViTs and Large Vision-Language Models (LVLMs) demonstrate that our approach consistently outperforms state-of-the-art pruning methods across diverse benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。