通过识别视觉标记相似性拐点，动态剪枝降低多模态大模型计算开销。
Sparsity Meets Similarity: Leveraging Long-Tail Distribution for Dynamic Optimized Token Representation in Multimodal Large Language Models
- 基于视觉标记与CLS token相似性的长尾分布，动态定位剪枝拐点。
- 仅用原22%的标记量，性能接近原始模型。
- 适合追求高效推理的多模态模型部署场景。
近期，多模态大语言模型（MM-LLMs）在各类任务中取得显著进展，但其高计算成本限制了广泛应用。主要计算负担源于在大语言模型层处理拼接的文本与视觉标记，输入标记长度直接影响效率。我们分析发现，视觉标记与CLS token的相似性呈长尾分布，仅有少数具有高相似性。为此，我们提出一种动态剪枝算法，通过识别视觉CLS token相似性曲线的拐点，有效裁剪视觉标记以加速模型性能。此外，在大语言模型层进行第二轮剪枝，通过视觉与文本特征交互过滤低相关性标记。实验表明，该方法在仅使用原标记量22%的情况下，性能仍与原始模型相当。代码将在接受后公开。
原文摘要 · Abstract (English)
Recently, multimodal large language models (MM-LLMs) have achieved significant success in various tasks, but their high computational costs limit widespread application. The main computational burden arises from processing concatenated text and visual tokens in the LLM layer, where input token length directly affects efficiency. Our analysis of visual tokens reveals that their similarity to the CLS token follows a long-tail distribution, with only a few showing high similarity. To address this, we propose a dynamic pruning algorithm that identifies the inflection point in the visual CLS token similarity curve, enabling effective trimming of visual markers to accelerate model performance. Additionally, we perform a second round of pruning in the LLM layer, filtering out low-correlation tokens through the interaction between visual and textual features. Experimental results demonstrate that our method achieves performance comparable to the original while utilizing only 22% of the original token quantity. Our source code will be made publicly available upon acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。