提出一种兼顾重要性和多样性的视觉令牌选择方法,显著提升大模型推理效率。
ID-Selection: Importance-Diversity Based Visual Token Selection for Efficient LVLM Inference
- 通过重要性评分与多样性抑制的迭代选择,统一处理冗余与信息丢失问题
- 在97.2%令牌剪枝下仅保留16个令牌,性能损失小于8.2%,推理算力降低超97%
- 无需额外训练,适用于多种主流视觉语言模型,适合高效部署场景
近期研究探索了视觉令牌剪枝以加速大视觉语言模型(LVLM)的推理。然而,现有方法常难以平衡令牌的重要性与多样性:基于重要性的方法易保留冗余令牌,而基于多样性的方法可能忽略关键信息。这一权衡在高剪枝率下尤为严重,此时仅保留少量视觉令牌至关重要。为此,我们提出ID-Selection,一种简单但高效的LVLM推理令牌选择策略。核心思想是将重要性估计与多样性感知的迭代选择相结合:先为每个令牌分配重要性分数,随后逐个选择高分令牌,并逐步抑制相似令牌的得分。如此,ID-Selection在统一过程中既保留了信息丰富的令牌,又减少了冗余。在5种LVLM主干网络和16个主流基准上的广泛实验表明,该方法在极端剪枝率下仍能持续实现更优的性能与效率。例如,在LLaVA-1.5-7B上,ID-Selection可剪除97.2%的视觉令牌(仅保留16个),推理计算量减少超过97%,同时保持91.8%的原始性能,且无需额外训练。
原文摘要 · Abstract (English)
Recent advances have explored visual token pruning to accelerate the inference of large vision-language models (LVLMs). However, existing methods often struggle to balance token importance and diversity: importance-based methods tend to retain redundant tokens, whereas diversity-based methods may overlook informative ones. This trade-off becomes especially problematic under high reduction ratios, where preserving only a small subset of visual tokens is critical. To address this issue, we propose ID-Selection, a simple yet effective token selection strategy for efficient LVLM inference. The key idea is to couple importance estimation with diversity-aware iterative selection: each token is first assigned an importance score, after which high-scoring tokens are selected one by one while the scores of similar tokens are progressively suppressed. In this way, ID-Selection preserves informative tokens while reducing redundancy in a unified selection process. Extensive experiments across 5 LVLM backbones and 16 main benchmarks demonstrate that ID-Selection consistently achieves superior performance and efficiency, especially under extreme pruning ratios. For example, on LLaVA-1.5-7B, ID-Selection prunes 97.2% of visual tokens, retaining only 16 tokens, while reducing inference FLOPs by over 97% and preserving 91.8% of the original performance, all without additional training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。