arXiv:2606.18681cs.CV2026-06被引 1

提出新方法在不训练情况下高效剪裁视觉令牌,提升多技能推理能力。

Moving Beyond Diversity: Visual Token Pruning as Subspace Reconstruction for Efficient VLMs

论文配图:Moving Beyond Diversity: Visual Token Pruning as Subspace Reconstruction for Efficient VLMs
图 1 · 摘自论文原文
  • 将令牌剪裁重构成子空间重建问题,基于投影残差迭代选择
  • 在LLaVA上剪裁94%令牌仍保留95%性能,复合任务提升显著
  • 发现低相关性令牌更保上下文,引入反相关性增强选择

尽管视觉语言模型(VLMs)表现优异,但其大量视觉令牌带来巨大计算开销。现有基于余弦相似度的多样性剪裁方法忽略幅度信息,难以准确还原原始特征,尤其在组合式多技能推理任务中表现不佳。本文提出SPARE,将令牌剪裁重构为列子集选择问题,通过迭代选取投影残差大的令牌实现重建驱动剪裁。我们发现反直觉现象:图像-文本相关性较低的令牌反而更利于保持上下文信息,并将其引入作为额外选择标准。在多个VLM和基准上的实验表明,SPARE持续达到最优性能,尤其在组合任务中优势明显。应用于LLaVA时,可移除高达94%的视觉令牌,同时保留95%的基线性能,且完全无需训练。

原文摘要 · Abstract (English)

Despite their remarkable performance, Vision Language Models (VLMs) incur substantial computational overhead due to the large number of visual tokens. While diversity maximization has become a dominant strategy for token reduction, existing methods rely on cosine-based normalized similarity that discards magnitude information, failing to faithfully approximate the original feature representation and leading to suboptimal performance, particularly on compositional multi-skill reasoning tasks. In this paper, we introduce SPARE, a subspace reconstruction method that reformulates token pruning as a column subset selection problem and explicitly minimizes reconstruction error. By iteratively selecting tokens with large projection residuals, SPARE performs reconstruction-driven pruning beyond angular diversity. Moreover, we reveal a counterintuitive anti-relevance phenomenon: tokens with lower image-text relevance score can better preserve contextual information. Based on this finding, we incorporate anti-relevance into SPARE as an additional selection criterion to promote context-aware token selection. Extensive experiments across multiple VLMs and benchmarks demonstrate that SPARE consistently achieves state-of-the-art performance, with strong gains on compositional tasks. When applied to LLaVA, SPARE removes up to 94% of visual tokens while retaining 95% of the baseline performance, all in a fully training-free manner.

视觉语言模型令牌剪裁子空间重建无训练优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。