通过分析视觉令牌的语义演变,实现高效无训练的模型压缩。
TransPrune: Token Transition Pruning for Efficient Large Vision-Language Model
- 基于令牌转移变化与指令注意力双重信号评估重要性
- 在8个基准上保持原模型性能,推理计算量减半以上
- 无需训练,适合部署在资源受限的多模态系统
大型视觉语言模型(LVLMs)在多模态学习中取得进展,但因大量视觉令牌导致计算开销高,亟需令牌剪枝提升推理效率。现有方法多依赖注意力机制判断令牌重要性,存在位置偏差等局限。本文提出一种新视角:基于令牌表示的转移行为捕捉语义信息。为此,我们设计TransPrune,一种无需训练的高效剪枝方法,通过组合令牌转移变化(TTV,衡量表示幅度与方向的变化)和指令引导注意力(IGA,衡量指令对图像令牌的关注程度)来评估重要性。大量实验表明,TransPrune在多个基准(如LLaVA-v1.5和LLaVA-Next)上表现接近原始模型,同时将推理计算量(TFLOPs)降低超过50%。值得注意的是,仅使用TTV即可达到与注意力方法相当的效果,且不依赖注意力机制。代码将在论文录用后公开于https://github.com/liaolea/TransPrune。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) have advanced multimodal learning but face high computational costs due to the large number of visual tokens, motivating token pruning to improve inference efficiency. The key challenge lies in identifying which tokens are truly important. Most existing approaches rely on attention-based criteria to estimate token importance. However, they inherently suffer from certain limitations, such as positional bias. In this work, we explore a new perspective on token importance based on token transitions in LVLMs. We observe that the transition of token representations provides a meaningful signal of semantic information. Based on this insight, we propose TransPrune, a training-free and efficient token pruning method. Specifically, TransPrune progressively prunes tokens by assessing their importance through a combination of Token Transition Variation (TTV)-which measures changes in both the magnitude and direction of token representations-and Instruction-Guided Attention (IGA), which measures how strongly the instruction attends to image tokens via attention. Extensive experiments demonstrate that TransPrune achieves comparable multimodal performance to original LVLMs, such as LLaVA-v1.5 and LLaVA-Next, across eight benchmarks, while reducing inference TFLOPs by more than half. Moreover, TTV alone can serve as an effective criterion without relying on attention, achieving performance comparable to attention-based methods. The code will be made publicly available upon acceptance of the paper at https://github.com/liaolea/TransPrune.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。