通过融合位置与相似性信息，动态优化视觉令牌剪枝。
Multi-Cue Adaptive Visual Token Pruning for Large Vision-Language Models
- 结合注意力、空间位置和令牌相似性，自适应评估重要性。
- 在多种模型与数据集上，剪枝后性能优于现有方法。
- 无需训练、可即插即用，适合高效部署大型视觉语言模型。
随着大型视觉语言模型（LVLMs）计算需求的增长，视觉令牌剪枝已被证明能有效提升推理速度和内存效率。传统剪枝方法主要依赖注意力分数判断令牌相关性，忽略了空间位置和令牌相似性等关键因素。为此，我们提出 AdaptPrune，一种无需训练的即插即用剪枝方法，基于注意力机制，引入空间距离与令牌相似性，并采用自适应非极大值抑制（NMS）策略。该方法建立在对大模型中观察到的现象之上：模型图像注意力存在位置偏差，且先前方法忽视了令牌信息冗余。通过融合注意力、空间和相似性信息，本方法实现对令牌重要性的全面评估，显著改进剪枝决策。我们在多个 LVLM 和基准测试中广泛验证了该方法的鲁棒性和适应性。结果表明，AdaptPrune 在不同剪枝比例下均持续优于现有方法。代码已开源：https://github.com/bzluan/AdaptPrune。
原文摘要 · Abstract (English)
As the computational needs of Large Vision-Language Models (LVLMs) increase, visual token pruning has proven effective in improving inference speed and memory efficiency. Traditional pruning methods in LVLMs predominantly focus on attention scores to determine token relevance, overlooking critical aspects such as spatial position and token similarity. To this end, we introduce AdaptPrune, a novel plug-and-play training-free pruning method that builds on conventional attention-based pruning by integrating spatial distance and token similarity with an adaptive NMS approach. Our method is based on several observed phenomena in large models: the positional bias in the model's image attention and the redundancy of token information ignored by previous approaches. By integrating attention, spatial, and similarity information, our approach ensures a comprehensive evaluation of token importance and substantially refines the pruning decisions. Our method has been extensively tested across various LVLMs and benchmarks, confirming its robustness and adaptability. The results demonstrate that AdaptPrune consistently outperforms existing methods across various pruning ratios. Code is available at https://github.com/bzluan/AdaptPrune.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。