arXiv:2509.01552cs.CV2025-09中稿 · CVPR被引 22

通过识别视觉令牌的冗余变化,动态删减无用信息,加速大模型推理。

Variation-aware Vision Token Dropping for Faster Large Vision-Language Models

  • 基于视觉令牌变化度动态删除低变动内容,避免位置偏差。
  • 图像和视频任务性能分别保持94.0%和98.6%,推理延迟降低31.5%与74.2%。
  • 适合需要高效推理的多模态应用,如实时视频理解与生成。

大型视觉语言模型(LVLMs)在多模态理解任务中表现卓越,但高分辨率图像与长视频理解导致令牌数量激增,显著降低推理效率。令牌压缩可通过减少处理量直接提升计算效率,无需改变模型结构。我们通过大量分析发现,现有内部语言模型的令牌压缩方法存在位置偏差和与高效算子不兼容两大关键问题,严重阻碍其在LVLM加速中的实际部署。本文首次从动态令牌变化角度提出方案,揭示视觉令牌在大模型中的变化具有任务无关特性。我们提出变异性感知视觉令牌丢弃(V²Drop),在推理过程中逐步移除变化极小的视觉令牌,从而提升计算效率。在多个模型与基准测试上的实验证明,该方法在图像和视频理解任务中分别保持94.0%和98.6%的原始性能,同时将语言模型生成延迟降低31.5%和74.2%。

原文摘要 · Abstract (English)

Large vision-language models (LVLMs) have demonstrated remarkable capabilities in multimodal understanding tasks. However, the increasing demand for high-resolution image and long-video understanding results in substantial token counts, consequently leading to reduced inference efficiency. Token compression offers a direct solution by reducing the number of tokens to be processed, thereby improving computational efficiency without architectural changes. Through extensive analysis, we identify two critical limitations in existing inner-LLM token compression methods: positional bias and incompatibility with efficient operators, which critically hinder their practical deployment for LVLM acceleration. This paper presents the first approach from a dynamic token variation perspective, revealing that visual token variations within LLMs exhibit task-agnostic properties. We propose Variation-aware Vision Token Dropping (\textit{i.e.}, \textbf{V$^2$Drop}), which progressively removes visual tokens with minimal variation during LVLM inference, thereby enhancing computational efficiency. Extensive experiments across multiple models and benchmarks consistently demonstrate that V$^2$Drop maintains \textbf{94.0\%} and \textbf{98.6\%} of the original performance for image and video understanding tasks respectively, while reducing LLM generation latency by \textbf{31.5\%} and \textbf{74.2\%}.

视觉语言模型令牌压缩推理加速多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。