提出IDPruner,让视觉令牌剪枝同时兼顾重要性和多样性。
IDPruner: Harmonizing Importance and Diversity in Visual Token Pruning for MLLMs
- 用MMR算法平衡重要性与语义多样性,实现最优剪枝。
- 在90%剪枝下仍保持86.40%性能,75%剪枝保留95.18%表现。
- 无需注意力图,兼容FlashAttention,适合高效部署。
多模态大语言模型(MLLMs)虽能力出色,但因视觉令牌数量庞大而面临严重计算瓶颈。视觉令牌剪枝通过大幅减少令牌数成为加速推理的关键技术。现有方法聚焦于令牌重要性、多样性或直观组合,缺乏两者的系统性融合框架。为此,我们首先系统分析重要性与语义多样性的权衡关系。基于此,提出重要性与多样性剪枝器(IDPruner),采用最大边际相关性(MMR)算法实现两者之间的帕累托最优平衡。关键在于,该方法无需依赖注意力图,完全兼容FlashAttention,支持一次性剪枝和高效部署。我们在多种模型架构和多模态基准上进行了广泛实验,结果表明IDPruner达到当前最优性能,并在不同架构与任务间具备优异泛化能力。特别地,在Qwen2.5-VL-7B-Instruct上,当剪枝75%令牌时仍保持95.18%基线性能,极端90%剪枝下仍维持86.40%。代码已开源。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities, yet they encounter significant computational bottlenecks due to the massive volume of visual tokens. Consequently, visual token pruning, which substantially reduces the token count, has emerged as a critical technique for accelerating MLLM inference. Existing approaches focus on token importance, diversity, or an intuitive combination of both, without a principled framework for their optimal integration. To address this issue, we first conduct a systematic analysis to characterize the trade-off between token importance and semantic diversity. Guided by this analysis, we propose the \textbf{I}mportance and \textbf{D}iversity Pruner (\textbf{IDPruner}), which leverages the Maximal Marginal Relevance (MMR) algorithm to achieve a Pareto-optimal balance between these two objectives. Crucially, our method operates without requiring attention maps, ensuring full compatibility with FlashAttention and efficient deployment via one-shot pruning. We conduct extensive experiments across various model architectures and multimodal benchmarks, demonstrating that IDPruner achieves state-of-the-art performance and superior generalization across diverse architectures and tasks. Notably, on Qwen2.5-VL-7B-Instruct, IDPruner retains 95.18\% of baseline performance when pruning 75\% of the tokens, and still maintains 86.40\% even under an extreme 90\% pruning ratio. Our code is available at https://github.com/Tencent/AngelSlim.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。