通过兼顾显著性与覆盖度,高效剪裁视觉令牌提升多模态模型性能。
SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs
- 基于显著性与令牌间关系的覆盖度联合建模。
- 在多个基准上优于现有方法,保持语义完整性。
- 适合需要高效推理的多模态大模型应用。
多模态大语言模型通常处理大量视觉令牌,造成显著计算开销,尽管其中许多是冗余的。现有视觉令牌剪枝方法主要依据注意力得分选择最显著的令牌,导致所选令牌语义不完整。本文提出一种新型视觉令牌剪枝策略——面向显著性与覆盖度的高效多模态大模型剪枝(SCOPE),通过联合建模所选令牌的显著性与覆盖度,更好保留语义完整性。具体地,我们基于令牌间关系定义集合覆盖度,并为每个未选令牌计算其覆盖增益,量化加入后能带来的额外覆盖范围。将显著性分数融入覆盖增益,构建SCOPE评分,迭代选择最高分令牌。我们在LLaVA-1.5和LLaVA-Next模型上于多个视觉语言理解基准进行广泛实验,结果表明本方法持续优于先前方法。代码已公开于https://github.com/kinredon/SCOPE。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) typically process a large number of visual tokens, leading to considerable computational overhead, even though many of these tokens are redundant. Existing visual token pruning methods primarily focus on selecting the most salient tokens based on attention scores, resulting in the semantic incompleteness of the selected tokens. In this paper, we propose a novel visual token pruning strategy, called \textbf{S}aliency-\textbf{C}overage \textbf{O}riented token \textbf{P}runing for \textbf{E}fficient MLLMs (SCOPE), to jointly model both the saliency and coverage of the selected visual tokens to better preserve semantic completeness. Specifically, we introduce a set-coverage for a given set of selected tokens, computed based on the token relationships. We then define a token-coverage gain for each unselected token, quantifying how much additional coverage would be obtained by including it. By integrating the saliency score into the token-coverage gain, we propose our SCOPE score and iteratively select the token with the highest SCOPE score. We conduct extensive experiments on multiple vision-language understanding benchmarks using the LLaVA-1.5 and LLaVA-Next models. Experimental results demonstrate that our method consistently outperforms prior approaches. Our code is available at \href{https://github.com/kinredon/SCOPE}{https://github.com/kinredon/SCOPE}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。