用跨模态互信息精简视觉令牌，提升多模态大模型推理效率
MI-Pruner: Crossmodal Mutual Information-guided Token Pruner for Efficient MLLMs
- 基于视觉与文本特征间的互信息评估令牌重要性，不依赖注意力机制
- 在多个数据集上实现更优压缩率与更低延迟，性能优于传统注意力剪枝方法
- 无需修改模型结构，适配性强，适合追求高效推理的工程部署场景
多模态大语言模型中，视觉信息相对文本稀疏，因此视觉令牌剪枝成为提升推理效率的关键。现有方法通常依据视觉编码器或语言模型解码器中的注意力分数来衡量令牌重要性，保留高分令牌并剪除其余。本文提出一种更精准的方法：在视觉与文本特征交互前，直接计算二者间的互信息（MI），以显式度量跨模态依赖关系。该方法不依赖特定机制信号，无需访问内部注意力图或修改网络结构，简单高效且无侵入性。实验表明，相比以往基于注意力的剪枝方法，本方法在保持性能的同时显著降低延迟。
原文摘要 · Abstract (English)
For multimodal large language models (MLLMs), visual information is relatively sparse compared with text. As a result, research on visual pruning emerges for efficient inference. Current approaches typically measure token importance based on the attention scores in the visual encoder or in the LLM decoder, then select visual tokens with high attention scores while pruning others. In this paper, we pursue a different and more surgical approach. Instead of relying on mechanism-specific signals, we directly compute Mutual Information (MI) between visual and textual features themselves, prior to their interaction. This allows us to explicitly measure crossmodal dependency at the feature levels. Our MI-Pruner is simple, efficient and non-intrusive, requiring no access to internal attention maps or architectural modifications. Experimental results demonstrate that our approach outperforms previous attention-based pruning methods with minimal latency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。