arXiv:2505.22654cs.CVcs.CL2025-05中稿 · TMLR 2026被引 41

提出VScan框架,高效压缩视觉令牌,显著提速大模型推理。

VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Models

  • 分两阶段压缩视觉令牌:编码时融合全局与局部扫描,解码时在中间层剪枝。
  • 在LLaVA-NeXT-7B上实现2.91倍预填充加速,浮点运算量减少10倍。
  • 适合追求高效部署的视觉语言模型研究者与工程师使用。

近期大型视觉语言模型(LVLMs)通过引入更细粒度的视觉感知与编码,提升了多模态理解能力,但由此带来的长视觉令牌序列导致计算开销巨大,难以实现实时部署。现有方法通常在视觉编码器输出层或语言模型早期层进行视觉令牌剪枝。本文通过系统实证研究视觉令牌在视觉编码与语言解码全过程中的处理方式,重新评估了这些设计的有效性。基于新洞察,提出VScan——一种两阶段视觉令牌压缩框架:(1) 在视觉编码阶段整合全局与局部扫描并结合令牌合并;(2) 在语言模型中间层引入剪枝机制。在四个LVLM上的广泛实验验证了VScan的有效性,在十六个基准上优于当前最优方法。尤其在LLaVA-NeXT-7B上,预填充速度提升2.91倍,浮点运算量减少10倍,性能保留率达95.4%。代码已开源。

原文摘要 · Abstract (English)

Recent Large Vision-Language Models (LVLMs) have advanced multi-modal understanding by incorporating finer-grained visual perception and encoding. However, such methods incur significant computational costs due to longer visual token sequences, posing challenges for real-time deployment. To mitigate this, prior studies have explored pruning unimportant visual tokens either at the output layer of the visual encoder or at the early layers of the language model. In this work, we revisit these design choices and reassess their effectiveness through comprehensive empirical studies of how visual tokens are processed throughout the visual encoding and language decoding stages. Guided by these insights, we propose VScan, a two-stage visual token reduction framework that addresses token redundancy by: (1) integrating complementary global and local scans with token merging during visual encoding, and (2) introducing pruning at intermediate layers of the language model. Extensive experimental results across four LVLMs validate the effectiveness of VScan in accelerating inference and demonstrate its superior performance over current state-of-the-arts on sixteen benchmarks. Notably, when applied to LLaVA-NeXT-7B, VScan achieves a 2.91$\times$ speedup in prefilling and a 10$\times$ reduction in FLOPs, while retaining 95.4\% of the original performance. Code is available at https://github.com/Tencent/SelfEvolvingAgent/tree/main/VScan.

视觉语言模型令牌压缩推理加速

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。