提出分区域动态选票的视觉令牌剪枝方法,提升多模态模型效率。
GridPrune: From "Where to Look" to "What to Select" in Visual Token Pruning for MLLMs
- 先按文本引导分配各区域剪枝名额,再局部精选,模拟人类视觉注意力机制
- 在LLaVA-NeXT-7B上仅用11.1%令牌保留96.98%性能,优于基线2.34%
- 适合追求高效率多模态推理、关注空间注意力分布的研究者
多模态大语言模型(MLLMs)在视觉语言任务中表现卓越,但大量视觉令牌带来显著计算开销。视觉令牌剪枝成为提升效率的关键技术。认知科学表明,人类先确定注视区域(‘何处看’),再选择具体元素(‘选什么’),实现粗粒度到细粒度的注意力分配。现有剪枝方法多直接优化‘选什么’,依赖注意力分数或相似性,忽视‘何处看’,导致空间分配低效、位置偏差及冗余令牌留存。本文提出GridPrune,将全局Top-K机制替换为‘全局引导、局部选择’的分区策略:首先通过文本条件引导动态分配各空间区域的令牌预算;随后在每个预算区内进行局部选择。实验表明,GridPrune在多种MLLM架构上均表现优异。在LLaVA-NeXT-7B上,仅使用11.1%的视觉令牌即可保留96.98%的全模型性能,且在相同剪枝率下优于最优基线2.34%。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have shown remarkable capabilities in a wide range of vision-language tasks. However, the large number of visual tokens introduces significant computational overhead. To address this issue, visual token pruning has emerged as a key technique for enhancing the efficiency of MLLMs. In cognitive science, humans tend to first determine which regions of a scene to attend to ("where to look") before deciding which specific elements within those regions to process in detail ("what to select"). This two-stage strategy enables the visual system to efficiently allocate attention at a coarse spatial level before performing fine-grained selection. However, existing pruning methods primarily focus on directly optimizing "what to select", typically using attention scores or similarity metrics. They rarely consider "where to look", which has been shown to lead to inefficient spatial allocation, positional bias, and the retention of irrelevant or redundant tokens. In this paper, we propose GridPrune, a method that replaces the global Top-K mechanism with a "guide-globally, select-locally" zonal selection system. GridPrune splits the pruning process into two steps: first, it uses text-conditional guidance to dynamically allocate a token budget across spatial zones; and then, it performs local selection within each budgeted zone. Experimental results demonstrate that GridPrune achieves superior performance across various MLLM architectures. On LLaVA-NeXT-7B, GridPrune retains 96.98% of the full performance while using 11.1% of the tokens, outperforming the best-performing baseline by 2.34% at the same pruning rate.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。