早阶段剪枝视觉标记，大幅提升多模态大模型推理速度
EvoPrune: Early-Stage Visual Token Pruning for Efficient MLLMs
- 在视觉编码阶段直接剪枝，按相似性与重要性保留关键标记
- 视频任务中提速2倍，性能损失不足1%
- 适合对延迟敏感的多模态应用部署
多模态大语言模型（MLLMs）在视觉-语言任务中表现强劲，但在高分辨率图像和视频等复杂场景下，视觉标记数量呈指数增长，严重制约其推理效率。现有剪枝方法多在视觉编码后进行，忽略了编码阶段的巨大计算开销。为此，我们提出EvoPrune，一种面向MLLMs的早期视觉标记剪枝方法，直接在视觉编码过程中执行剪枝。EvoPrune采用分层剪枝策略，基于标记的相似性、多样性及注意力重要性，选择特定编码层保留最具信息量的视觉标记。在图像与视频基准上的大量实验验证了其有效性。尤其在VideoMME数据集上，EvoPrune实现2×推理加速，性能下降低于1%，展现出在低延迟多模态模型部署中的巨大潜力。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have shown strong performance in vision-language tasks, but their inference efficiency is severely limited by the exponential growth of visual tokens in complex scenarios such as high-resolution images and videos. Existing visual token pruning methods mainly operate after visual encoding, overlooking the substantial computational cost incurred during the encoding stage. To address this issue, we propose EvoPrune, an early-stage visual token pruning method for MLLMs that performs pruning directly during visual encoding. Specifically, EvoPrune employs a layer-wise pruning strategy guided by token similarity, diversity, and attention-based importance to retain the most informative visual tokens at selected encoding layers. Extensive experiments on image and video benchmarks validate the effectiveness of EvoPrune. In particular, on the VideoMME dataset, EvoPrune achieves 2$\times$ inference speedup with less than 1% performance degradation, demonstrating its potential for latency-sensitive MLLM deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。