arXiv:2604.23950cs.CV2026-04中稿 · ICLR被引 6

提出LearnPruner框架,高效压缩视觉语言模型的冗余信息。

LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models

论文配图:LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models
图 1 · 摘自论文原文
  • 分两阶段剪枝:先用可学习模块删减视觉编码器冗余帧,再在LLM中间层保留关键语义帧。
  • 仅用5.5%视觉令牌即可保持95%原模型性能,推理速度提升3.2倍。
  • 适合追求高效视觉理解的部署场景,尤其适用于长视频输入任务。

视觉语言模型(VLMs)在视觉理解与推理方面表现出色,但因输入序列过长带来巨大计算开销。现有方法通过剪枝不重要视觉令牌缓解问题,其核心在于判断令牌重要性,当前多依赖视觉编码器或大语言模型(LLM)的注意力得分。本文分析发现,视觉编码器存在注意力下沉现象,难以聚焦有信息量的前景区域;而尽管LLM中存在位置注意力偏差,文本到视觉注意力在中间层仍具抗偏差能力,能有效指导剪枝。基于此,我们提出两阶段剪枝框架LearnPruner:首先在视觉编码器后使用可学习模块移除冗余视觉令牌,再在LLM中间层保留任务相关令牌。实验表明,LearnPruner在仅使用5.5%视觉令牌时仍能保持约95%原始性能,并实现3.2倍推理加速,展现出优越的精度-效率平衡。

原文摘要 · Abstract (English)

Vision-Language Models (VLMs) have recently demonstrated remarkable capabilities in visual understanding and reasoning, but they also impose significant computational burdens due to long visual sequence inputs. Recent works address this issue by pruning unimportant visual tokens, achieving substantial computational reduction while maintaining model performance. The core of token pruning lies in determining token importance, with current approaches primarily relying on attention scores from vision encoders or Large Language Models (LLMs). In this paper, we analyze the effectiveness of attention mechanisms in both vision encoders and LLMs. We find that vision encoders suffer from attention sink, leading to poor focus on informative foreground regions, while in LLMs, although prior studies have identified attention bias toward token positions, text-to-vision attention demonstrates resistance to this bias and enables effective pruning guidance in middle layers. Based on these observations, we propose LearnPruner, a two-stage token pruning framework that first removes redundant vision tokens via a learnable pruning module after the vision encoder, then retains only task-relevant tokens in the LLM's middle layer. Experimental results show that our LearnPruner can preserve approximately 95% of the original performance while using only 5.5% of vision tokens, and achieve 3.2$\times$ inference acceleration, demonstrating a superior accuracy-efficiency trade-off.

视觉语言模型注意力剪枝高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。