通过构建最优保留集,高效剪枝视觉令牌,提升多模态大模型推理速度。
TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference

- 基于信息论从头设计剪枝原则,兼顾任务相关性与语义多样性。
- 在LLaVA-NeXT上剪掉77.8%的视觉令牌,性能反而提升0.6%。
- 无需训练、适配多种模型,适合追求轻量化部署的研究者。
多模态大语言模型(MLLM)虽具备强大跨模态推理能力,但受限于大量视觉令牌带来的计算开销。现有视觉令牌剪枝方法存在缺陷:基于注意力的准则常保留冗余令牌,基于多样性的准则又忽视用户指令。即使融合多准则的方法也缺乏内在目标的严格定义。本文从第一性原理出发,将剪枝问题建模为构建“视觉令牌最优保留集”。通过自上而下的信息论分析,提出任务相关性、信息覆盖度和语义多样性三大核心原则。据此提出无训练、模型无关的剪枝模块TOPS,可应用于7种不同MLLM主干网络。在14个基准上的实验表明,其在多种剪枝率下均优于已有方法。尤其在LLaVA-NeXT模型上,剪除77.8%视觉令牌后,7B与13B版本性能分别保持100.0%与100.6%,表明剪枝能缓解幻觉,为轻量级MLLM设计提供新思路。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have achieved strong multimodal reasoning capabilities, but their efficiency is limited by the large number of visual tokens, which introduces substantial computational overhead. Visual token pruning offers a natural solution, yet existing methods are imperfect: attention-based criteria tend to retain redundant tokens, while diversity-based criteria are often agnostic to user instructions. Even methods that combine multiple criteria still lack a principled formulation of the intrinsic objective of token pruning. In this paper, we revisit visual token pruning from a first-principles perspective and formulate it as constructing Token Optimal Preservation Sets. Through a top-down information-theoretic analysis, we identify three fundamental principles for effective token selection: Task Relevance, Information Coverage, and Semantic Diversity. Based on these principles, we propose TOPS, a training-free and model-agnostic pruning module that can be applied to various MLLMs. Extensive experiments on 7 MLLM backbones and 14 benchmarks demonstrate that TOPS outperforms prior methods under diverse pruning settings. Notably, on LLaVA-NeXT, TOPS removes 77.8% of visual tokens while preserving 100.0% and 100.6% performance on its 7B and 13B models, respectively, suggesting that pruning redundant visual tokens can sometimes mitigate hallucination and inspire future lightweight MLLM design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。