提出一种无需训练的视觉令牌剪枝方法,兼顾冗余与空间分布。
VLM-Pruner: Buffering for Spatial Sparsity in an Efficient VLM Centrifugal Token Pruning Paradigm
- 基于离心式选择机制,从近到远保留关键视觉令牌。
- 在88.9%剪枝率下仍保持模型性能,推理速度显著提升。
- 适合移动端部署,尤其对资源受限场景友好。
视觉语言模型(VLMs)在图像理解任务中表现优异,但大量视觉令牌带来显著计算开销,制约其在移动设备上的部署。现有剪枝方法仅依赖令牌重要性,忽略令牌间的冗余关系,导致重复令牌保留过多,浪费计算资源。部分冗余感知方法虽考虑冗余,却忽视视觉令牌的空间关系,造成保留令牌过于稀疏,难以覆盖目标物体区域。为此,本文提出VLM-Pruner,一种无需训练的令牌剪枝算法,显式平衡冗余与空间稀疏性。引入离心式令牌剪枝范式,实现由近及远的选择,优先保留细粒度物体细节。设计空间稀疏缓冲(BSS)准则,延迟对空间远端令牌的选择。采用并行贪心策略高效完成令牌筛选。为缓解剪枝带来的信息损失,选择性将被丢弃令牌中的显著信息融合至保留令牌中。全面对比表明,VLM-Pruner在五种VLM上均优于强基线,在88.9%剪枝率下保持性能,并实现端到端推理加速。代码已开源。
原文摘要 · Abstract (English)
Vision-language models (VLMs) excel at image understanding tasks, but the large number of visual tokens imposes significant computational costs, hindering deployment on mobile devices. Many pruning methods rely solely on token importance and thus overlook inter-token redundancy, retaining numerous duplicated tokens and wasting capacity. Although some redundancy-aware approaches have been proposed, they often ignore the spatial relationships among visual tokens. This can lead to overly sparse selections of retained tokens that fail to adequately cover the regions of target objects. To address these limitations, we propose VLM-Pruner, a training-free token pruning algorithm that explicitly balances redundancy and spatial sparsity. We introduce a centrifugal token pruning paradigm that enables near-to-far selection while prioritizing the preservation of fine-grained object details. Moreover, we design a Buffering for Spatial Sparsity (BSS) criterion that defers the selection of spatially distant tokens. We further adopt a parallel greedy strategy to conduct token selection efficiently. To mitigate information loss from pruning, we selectively fuse salient information from the discarded tokens into the retained ones. Comprehensive comparisons demonstrate that VLM-Pruner consistently outperforms strong baselines across five VLMs with an 88.9\% pruning rate, while delivering an end-to-end inference speedup. The code is available at https://github.com/Casey-bit/VLMPruner.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。