arXiv:2505.22038cs.CVcs.AI2025-05NeurIPS被引 26

提出平衡剪枝方法,让视觉语言模型更快更省资源。

Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization

  • 分阶段剪枝:前期关注后续层影响,后期保本地输出一致
  • 压缩率78%时,性能损失仅3.3%,平均保留96.7%原始能力
  • 无需重训练,适配多种视觉语言模型,适合部署优化

大规模视觉语言模型(LVLM)在多模态任务中表现优异,但将图像编码为数千个视觉令牌导致显著计算开销,尤其在动态高分辨率输入下更为严重。现有剪枝方法通常基于注意力分数或令牌多样性选择保留项,但忽略了剪枝对当前层输出(局部)和后续层输出(全局)的联合影响,导致次优决策。为此,我们提出平衡令牌剪枝(BTP),一种即插即用的视觉令牌剪枝方法。该方法利用小型校准集将剪枝过程划分为多个阶段:早期阶段侧重于剪枝对深层输出的影响,后期则聚焦于保持局部输出一致性。在多种LVLM和基准测试上的实验表明,该方法具有广泛有效性。在平均性能保留96.7%的前提下,实现78%的令牌压缩率。代码已公开于 https://github.com/EmbodiedCity/NeurIPS2025-Balanced-Token-Pruning。

原文摘要 · Abstract (English)

Large Vision-Language Models (LVLMs) have shown impressive performance across multi-modal tasks by encoding images into thousands of tokens. However, the large number of image tokens results in significant computational overhead, and the use of dynamic high-resolution inputs further increases this burden. Previous approaches have attempted to reduce the number of image tokens through token pruning, typically by selecting tokens based on attention scores or image token diversity. Through empirical studies, we observe that existing methods often overlook the joint impact of pruning on both the current layer's output (local) and the outputs of subsequent layers (global), leading to suboptimal pruning decisions. To address this challenge, we propose Balanced Token Pruning (BTP), a plug-and-play method for pruning vision tokens. Specifically, our method utilizes a small calibration set to divide the pruning process into multiple stages. In the early stages, our method emphasizes the impact of pruning on subsequent layers, whereas in the deeper stages, the focus shifts toward preserving the consistency of local outputs. Extensive experiments across various LVLMs demonstrate the broad effectiveness of our approach on multiple benchmarks. Our method achieves a 78% compression rate while preserving 96.7% of the original models' performance on average. Our code is available at https://github.com/EmbodiedCity/NeurIPS2025-Balanced-Token-Pruning.

视觉语言模型令牌剪枝加速推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。