提出贪心剪枝法，同时保留视觉关键性和多样性，提升大模型推理效率。
GreedyPrune: Retenting Critical Visual Token Set for Large Vision Language Models
- 贪心算法联合优化语义重要性与视觉多样性，实现无训练剪枝。
- 在多个模型和任务上保持领先性能，推理延迟显著降低。
- 适合部署在资源受限设备上的多模态大模型加速应用。
尽管大视觉语言模型（LVLM）在图像理解任务中表现卓越，其计算效率仍是重大挑战，尤其在资源受限设备上，因需处理大量视觉标记而代价高昂。近期，无训练视觉标记剪枝方法因其低成本备受关注。然而，现有方法存在两大局限：基于语义显著性的策略主要关注高交叉注意力标记，常忽视视觉多样性；而基于视觉多样性的方法在高压缩比下可能误删语义重要标记。本文提出 GreedyPrune，一种无训练、即插即用的视觉标记剪枝算法，旨在联合优化语义显著性与视觉多样性。我们将剪枝过程形式化为组合优化问题，并证明贪心算法能有效平衡计算效率与模型精度。大量实验验证了该方法的有效性，结果显示，GreedyPrune 在多种多模态任务和模型上均达到最优性能，同时显著降低端到端推理延迟。
原文摘要 · Abstract (English)
Although Large Vision Language Models (LVLMs) have demonstrated remarkable performance in image understanding tasks, their computational efficiency remains a significant challenge, particularly on resource-constrained devices due to the high cost of processing large numbers of visual tokens. Recently, training-free visual token pruning methods have gained popularity as a low-cost solution to this issue. However, existing approaches suffer from two key limitations: semantic saliency-based strategies primarily focus on high cross-attention visual tokens, often neglecting visual diversity, whereas visual diversity-based methods risk inadvertently discarding semantically important tokens, especially under high compression ratios. In this paper, we introduce GreedyPrune, a training-free plug-and-play visual token pruning algorithm designed to jointly optimize semantic saliency and visual diversity. We formalize the token pruning process as a combinatorial optimization problem and demonstrate that greedy algorithms effectively balance computational efficiency with model accuracy. Extensive experiments validate the effectiveness of our approach, showing that GreedyPrune achieves state-of-the-art accuracy across various multimodal tasks and models while significantly reducing end-to-end inference latency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。