用语言引导剪枝视觉令牌，大幅降低多模态大模型计算开销。
LVPruning: An Effective yet Simple Language-Guided Vision Token Pruning Approach for Multi-modal Large Language Models
- 通过交叉注意力评估视觉令牌重要性，动态剪枝非关键部分。
- 可剪掉90%视觉令牌，推理耗能降低62.1%，性能仅降0.45%。
- 无需修改模型参数，适合部署在资源受限场景的工程师使用。
多模态大语言模型（MLLMs）通过融合视觉与文本模态取得了显著进展，但因处理大量视觉令牌导致计算开销巨大，限制了其在资源受限环境中的实用性。本文提出语言引导的视觉令牌剪枝方法（LVPruning），该方法利用交叉注意力模块基于视觉与语言令牌的交互计算视觉令牌重要性，决定哪些应被剪枝。重要的是，LVPruning无需修改原始模型参数，易于集成或移除。实验表明，LVPruning可在LLaVA-1.5模型中间层剪除高达90%的视觉令牌，使推理时的千兆浮点运算量（TFLOPs）下降62.1%，在九个多模态基准测试上平均性能损失仅为0.45%。
原文摘要 · Abstract (English)
Multi-modal Large Language Models (MLLMs) have achieved remarkable success by integrating visual and textual modalities. However, they incur significant computational overhead due to the large number of vision tokens processed, limiting their practicality in resource-constrained environments. We introduce Language-Guided Vision Token Pruning (LVPruning) for MLLMs, an effective yet simple method that significantly reduces the computational burden while preserving model performance. LVPruning employs cross-attention modules to compute the importance of vision tokens based on their interaction with language tokens, determining which to prune. Importantly, LVPruning can be integrated without modifying the original MLLM parameters, which makes LVPruning simple to apply or remove. Our experiments show that LVPruning can effectively reduce up to 90% of vision tokens by the middle layer of LLaVA-1.5, resulting in a 62.1% decrease in inference Tera Floating-Point Operations Per Second (TFLOPs), with an average performance loss of just 0.45% across nine multi-modal benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。