通过中心化修正视觉标记冗余剪枝,提升大模型效率与性能
Centering before Pruning: Lightweight Geometry Correction for Diversity-Based Visual Token Pruning in LVLMs

- 先对视觉标记中心化再计算相似度,增强多样性分辨力
- 保留原始空间的显著性偏好,避免剪枝丢失关键信息
- 轻量无额外开销,适配各类视觉语言模型和任务
大型视觉-语言模型(LVLM)因长而高度冗余的视觉标记序列导致推理成本高昂。基于多样性的剪枝方法通过成对余弦相似度选择标记子集以降低开销。我们发现,原始视觉标记间的相似度高度集中于正值区间,限制了区分非冗余标记的能力。一个自然的改进方式是在计算余弦相似度前对标记特征进行中心化。中心化确实揭示了更丰富的成对结构,但单独使用反而降低剪枝性能。我们发现,这种看似矛盾的现象源于原始几何结构不仅反映成对多样性,还隐式偏好全局独特的标记,这些标记通常包含语义信息。中心化虽更好解析子集多样性,却损失了这一有用的标记级偏好,揭示出多样性与独特性在原始几何中相互纠缠。基于此分析,我们提出轻量级的中心化几何剪枝器(Cen-Prune),在使用中心化余弦相似度衡量子集多样性的同时,保留原始空间的独特性作为互补的标记级偏好。该方法为即插即用的轻量修正,不改变底层选择机制,计算开销可忽略。在多个图像与视频理解基准及不同架构的LVLM上广泛实验表明,Cen-Prune可稳定提升现有基于多样性的剪枝器的整体性能。
原文摘要 · Abstract (English)
Large vision-language models (LVLMs) incur substantial inference costs due to their long and highly redundant visual-token sequences. Diversity-based pruning mitigates this cost by selecting token subsets based on pairwise cosine similarity. We find, however, that similarities between raw visual tokens are strongly concentrated in the positive range, limiting their ability to distinguish non-redundant tokens. A natural way to improve this resolution is to center token features before computing cosine similarity. Centering indeed reveals a substantially richer pairwise structure, yet unexpectedly degrades pruning performance when used alone. We show that this apparent contradiction arises because the raw geometry does more than represent pairwise diversity: it also implicitly favors globally distinctive tokens, which tend to contain semantically informative content. Centering better resolves subset diversity but loses this useful token-wise preference, revealing that diversity and distinctiveness are entangled in the raw geometry. Based on this analysis, we propose the \textbf{Cen}tered Geometry \textbf{Prune}r (Cen-Prune), which measures subset diversity using centered cosine similarity while retaining raw-space distinctiveness as a complementary token-wise preference. This lightweight, plug-and-play correction leaves the underlying selection mechanism unchanged and incurs negligible computational overhead. Extensive experiments across multiple image- and video-understanding benchmarks and LVLM architectures demonstrate that Cen-Prune provides robust improvements in overall performance across existing diversity-based pruners.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。