无需训练即可加速大模型，保持性能同时提升两倍速度。
freePruner: A Training-free Approach for Large Multimodal Model Acceleration
- 通过贡献度与注意力模式分析，分两阶段筛选关键视觉语言令牌。
- 在无训练条件下实现2倍加速，且在主流问答任务上表现接近原模型。
- 可与量化等后训练技术结合，适合部署高算力需求的多模态模型。
大型多模态模型（LMMs）在视觉-语言任务中表现出色，但因计算开销大而难以部署。现有令牌压缩方法通常需大量重训练或微调，对拥有专有训练数据的先进模型不实用。我们提出freePruner，一种无需训练的令牌压缩方法，可直接应用于任意开源LMM。不同于依赖令牌合并的方法，freePruner采用两阶段令牌选择策略：（1）利用自定义贡献度指标识别承载高层语义的关键令牌；（2）通过注意力模式分析选取保留低层视觉细节的互补令牌。大量实验表明，freePruner在无训练设置下实现2倍加速，且在主流视觉问答基准上性能相当。此外，该方法与后训练压缩技术（如量化）正交，可协同使用，为LMM高效部署提供可行方案。
原文摘要 · Abstract (English)
Large Multimodal Models (LMMs) have demonstrated impressive capabilities in visual-language tasks but face significant deployment challenges due to their high computational demands. While recent token reduction methods show promise for accelerating LMMs, they typically require extensive retraining or fine-tuning, making them impractical for many state-of-the-art models, especially those with proprietary training data. We propose freePruner, a training-free token reduction approach that can be directly applied to any open-source LMM without additional training. Unlike existing methods that rely heavily on token merging operations, freePruner employs a two-stage token selection strategy: (1) identifying pivotal tokens that capture high-level semantic information using our designed contribution degree metric, and (2) selecting complementary tokens that preserve essential low-level visual details through attention pattern analysis. Extensive experiments demonstrate that freePruner achieves 2x acceleration while maintaining comparable performance across mainstream visual question-answering benchmarks in the training-free setting. Moreover, freePruner is orthogonal to and can be combined with other post-training acceleration techniques, such as post-training quantization, providing a practical solution for efficient LMM deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。