arXiv:2603.09480cs.CV2026-03中稿 · ICLR被引 18

通过重要性与多样性协同压缩视觉令牌,显著提升多模态模型效率。

Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity

  • 分两阶段:聚类语义一致的视觉令牌,再组内去重保留关键代表
  • 仅用11.1%令牌达96.3%准确率,极端压缩下仍超前人2.5%
  • 无需训练,适配图像视频多模态,部署速度快7.8倍

视觉-语言模型(VLMs)因生成过多视觉令牌而面临严重计算低效。尽管先前研究指出大量视觉令牌冗余,现有压缩方法难以兼顾重要性保留与信息多样性。为此,我们提出PruneSID——一种无需训练的协同重要性-多样性压缩方法,包含两阶段流程:(1) 主要语义成分分析(PSCA),将令牌聚类为语义连贯组,确保概念覆盖全面;(2) 组内非最大值抑制(NMS),在保留每组关键代表性令牌的同时剔除冗余。此外,PruneSID引入基于图像复杂度的信息感知动态压缩比机制,优化不同场景下的平均信息保留效果。大量实验表明其性能领先:在LLaVA-1.5上仅保留11.1%令牌即达96.3%准确率,在LLaVA-NeXT上极端压缩至5.6%时仍保持92.8%准确率,优于先前方法2.5%,预填充速度提升7.8倍。该框架可泛化至多种VLM及图像、视频模态,展现强跨模态通用性。代码已开源:https://github.com/ZhengyaoFang/PruneSID。

原文摘要 · Abstract (English)

Vision-language models (VLMs) face significant computational inefficiencies caused by excessive generation of visual tokens. While prior work shows that a large fraction of visual tokens are redundant, existing compression methods struggle to balance importance preservation and information diversity. To address this, we propose PruneSID, a training-free Synergistic Importance-Diversity approach featuring a two-stage pipeline: (1) Principal Semantic Components Analysis (PSCA) for clustering tokens into semantically coherent groups, ensuring comprehensive concept coverage, and (2) Intra-group Non-Maximum Suppression (NMS) for pruning redundant tokens while preserving key representative tokens within each group. Additionally, PruneSID incorporates an information-aware dynamic compression ratio mechanism that optimizes token compression rates based on image complexity, enabling more effective average information preservation across diverse scenes. Extensive experiments demonstrate state-of-the-art performance, achieving 96.3% accuracy on LLaVA-1.5 with only 11.1% token retention, and 92.8% accuracy at extreme compression rates (5.6%) on LLaVA-NeXT, outperforming prior methods by 2.5% with 7.8 $\times$ faster prefilling speed compared to the original model. Our framework generalizes across diverse VLMs and both image and video modalities, showcasing strong cross-modal versatility. Code is available at https://github.com/ZhengyaoFang/PruneSID.

视觉压缩多模态效率优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。