提出无需重训练的双模块剪枝法，高效降低多模态大模型显存与延迟。
Script: Graph-Structured and Query-Conditioned Semantic Token Pruning for Multimodal Large Language Models
- 构建图结构与查询感知双重剪枝模块，动态保留相关视觉信息。
- 在LLaVA-NeXT-7B上实现6.8倍预填充加速、10倍计算量减少，性能保留96.88%。
- 适用于图像与视频理解任务，适配多种多模态大模型且无需微调。
多模态大语言模型（MLLMs）中视觉令牌数量激增导致内存消耗和推理延迟过高，尤其在处理高分辨率图像与视频时更为显著。现有令牌剪枝方法常忽略用户查询的相关性，或受限于注意力机制，适应性与有效性不足。为此，我们提出Script——一种无需重训练、可泛化至多种MLLMs的即插即用剪枝方法。Script包含两个模块：图结构剪枝模块用于移除视觉冗余令牌，查询感知语义剪枝模块则保留与查询相关的视觉信息。二者协同提升多模态任务表现。在14个图像与视频理解基准测试中，Script consistently优于现有剪枝方法。在LLaVA-NeXT-7B上，实现最高6.8倍预填充速度提升与10倍FLOP减少，同时保持96.88%的原始性能。
原文摘要 · Abstract (English)
The rapid growth of visual tokens in multimodal large language models (MLLMs) leads to excessive memory consumption and inference latency, especially when handling high-resolution images and videos. Token pruning is a technique used to mitigate this issue by removing redundancy, but existing methods often ignore relevance to the user query or suffer from the limitations of attention mechanisms, reducing their adaptability and effectiveness. To address these challenges, we propose Script, a plug-and-play pruning method that requires no retraining and generalizes across diverse MLLMs. Script comprises two modules: a graph-structured pruning module that removes visually redundant tokens, and a query-conditioned semantic pruning module that preserves query-relevant visual information. Together, they enhance performance on multimodal tasks. Experiments on fourteen benchmarks across image and video understanding tasks show that Script consistently achieves higher model efficiency and predictive accuracy compared to existing pruning methods. On LLaVA-NeXT-7B, it achieves up to 6.8x prefill speedup and 10x FLOP reduction, while retaining 96.88% of the original performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。