发现注意力头中仅少数关键头影响视觉标记选择，据此提出高效剪枝方法。
Not All Attention Heads Contribute to Critical Visual Token Selection: Head-Aware Pruning Matters More
- 按注意力头重要性分组，只保留关键头进行剪枝
- 在LLaVA-1.5-7B上实现88.9%剪枝率，性能损失仅4.1%
- 无需训练，可显著提升视觉语言模型推理速度
视觉语言模型（VLMs）在多种视觉任务中表现优异，但其视觉标记数量激增导致推理时内存与计算开销巨大。现有剪枝方法通常对大语言模型骨干网络中某层的所有注意力头的注意力分数求和后剪枝。本文揭示：识别关键视觉标记的能力集中在少数注意力头中，仅依赖这些头即可提升任务性能。为此，我们提出无需训练的渐进式视觉标记剪枝框架ProViP：先基于输入标记嵌入相似性去除冗余标记，再在推理过程中通过头感知剪枝进一步压缩。实验表明，ProViP在保持高精度的同时大幅提高效率。以LLaVA-1.5-7B为例，88.9%剪枝率下仍保留95.9%原始性能，推理速度提升1.62倍。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) have exhibited impressive performance across diverse visual scenarios. However, this success comes at the cost of explosive growth in visual tokens, which imposes substantial memory and computational overhead during inference, ultimately increasing latency. To improve VLM inference efficiency, a typical class of visual token pruning methods estimates token importance by aggregating attention scores across all heads in the pruning layer of the Large Language Model (LLM) backbone and prunes tokens based on aggregated scores. However, in this paper, we reveal a compelling phenomenon: the capability to pinpoint critical visual tokens is concentrated within a small fraction of heads. Aggregation exclusively on these heads can improve task performance. Inspired by this observation, we propose ProViP, a training-free progressive visual token pruning framework. ProViP first removes redundant visual tokens based on the embedding similarity of input tokens before reasoning of the LLM backbone, and then further prunes tokens during reasoning via head-aware pruning. Experiments demonstrate that ProViP delivers outstanding task performance and inference efficiency. For instance, when applied to LLaVA-1.5-7B, ProViP retains 95.9% of the original performance and achieves 1.62x inference speedup under an 88.9% pruning ratio.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。