提出按重复度剪枝视觉令牌，比重要性剪枝更高效且兼容加速算子。
Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters More
- 基于令牌间重复度选择关键片段，无需训练
- 可剪掉88.9%视觉令牌，性能几乎不变
- 适合追求推理加速的多模态模型部署者
多模态大模型中的视觉令牌因长度远超语言模态，导致巨大计算开销。现有方法通常通过定义令牌重要性进行剪枝，但本文发现重要性并非理想剪枝指标，其效果常劣于随机剪枝，且不兼容高效注意力算子。为此，我们提出DART（Duplication-Aware Reduction of Tokens），根据令牌与其它令牌的重复度进行剪枝：先选取少量枢纽令牌，再保留与枢纽重复度低的令牌，以最小化信息损失。实验表明，DART可剪除88.9%视觉令牌，在总时间上提速1.99倍，预填充阶段提速2.99倍，同时保持良好对高效注意力算子的兼容性。代码已开源。
原文摘要 · Abstract (English)
Vision tokens in multimodal large language models often dominate huge computational overhead due to their excessive length compared to linguistic modality. Abundant recent methods aim to solve this problem with token pruning, which first defines an importance criterion for tokens and then prunes the unimportant vision tokens during inference. However, in this paper, we show that the importance is not an ideal indicator to decide whether a token should be pruned. Surprisingly, it usually results in inferior performance than random token pruning and leading to incompatibility to efficient attention computation operators.Instead, we propose DART (Duplication-Aware Reduction of Tokens), which prunes tokens based on its duplication with other tokens, leading to significant and training-free acceleration. Concretely, DART selects a small subset of pivot tokens and then retains the tokens with low duplication to the pivots, ensuring minimal information loss during token pruning. Experiments demonstrate that DART can prune 88.9% vision tokens while maintaining comparable performance, leading to a 1.99$\times$ and 2.99$\times$ speed-up in total time and prefilling stage, respectively, with good compatibility to efficient attention operators. Our codes are available at https://github.com/ZichenWen1/DART.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。