通过剪枝与聚类减少视觉令牌，提升视觉语言模型推理速度。
PACT: Pruning and Clustering-Based Token Reduction for Faster Visual Language Models
- 用新重要性度量识别无意义视觉令牌，不依赖注意力分数。
- 提出距离有界密度峰值聚类算法，约束簇内元素距离。
- 兼容FlashAttention，显著降低推理时间和内存占用。
视觉语言模型因需额外输入令牌表示视觉信息而消耗大量计算资源。然而，这些视觉令牌常包含冗余和无关信息，导致令牌数量过多。为解决此问题，我们提出PACT方法，在语言模型早期层通过剪枝无关令牌并合并视觉上冗余的令牌，以降低推理时间与内存使用。该方法采用一种新颖的重要性度量来识别无意义令牌，无需依赖注意力分数，因此与FlashAttention兼容。同时，我们提出一种名为距离有界密度峰值聚类的新聚类算法，能在预设阈值下约束簇内元素距离，实现高效聚类。通过大量实验验证了PACT的有效性。
原文摘要 · Abstract (English)
Visual Language Models require substantial computational resources for inference due to the additional input tokens needed to represent visual information. However, these visual tokens often contain redundant and unimportant information, resulting in an unnecessarily high number of tokens. To address this, we introduce PACT, a method that reduces inference time and memory usage by pruning irrelevant tokens and merging visually redundant ones at an early layer of the language model. Our approach uses a novel importance metric to identify unimportant tokens without relying on attention scores, making it compatible with FlashAttention. We also propose a novel clustering algorithm, called Distance Bounded Density Peak Clustering, which efficiently clusters visual tokens while constraining the distances between elements within a cluster by a predefined threshold. We demonstrate the effectiveness of PACT through extensive experiments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。