通过隐式权重视角，实现无需训练的视觉语言模型令牌剪枝。
IWP: Token Pruning as Implicit Weight Pruning in Large Vision Language Models
- 将注意力机制视为隐式线性层，基于秩1更新选择最优令牌子集。
- 提出新度量同时评估令牌信息量与冗余度，提升剪枝精度。
- 无需训练，适用于高效推理，适合部署资源受限场景。
大型视觉语言模型在图像和视频理解任务中表现优异，但其计算开销随视觉令牌数量快速增加。现有令牌剪枝方法多采用经验性策略，忽视了注意力机制的内在机理。本文提出一种基于注意力对偶形式的新颖无训练剪枝框架。我们将注意力重构成一个隐式线性层，其权重矩阵为各令牌键值对生成的秩1外积之和。因此，令牌剪枝转化为选择一组最优秩1更新以逼近原始对偶权重矩阵。将该视角扩展至标准Softmax注意力后，推导出一种新度量，可同时量化令牌的信息量与信息冗余。为高效筛选最优子集，引入渐进式分块最大边际相关性（Progressive Chunked Maximal Marginal Relevance）。大量实验表明，该方法在性能与效率间取得更优平衡，并为现有剪枝方法提供了新视角。
原文摘要 · Abstract (English)
Large Vision Language Models show impressive performance across image and video understanding tasks, yet their computational cost grows rapidly with the number of visual tokens. Existing token pruning methods mitigate this issue through empirical approaches while overlooking the internal mechanism of attention. In this paper, we propose a novel training free token pruning framework grounded in the dual form perspective of attention. We reformulate attention as an implicit linear layer whose weight matrix is the sum of rank 1 outer products, each generated by a single token's key value pair. Token pruning thus reduces to selecting an optimal subset of these rank 1 updates that best approximates the original dual weight matrix. Extending this perspective to standard softmax attention in LVLMs, we derive a novel metric quantifying both a token's information magnitude and information duplication. To efficiently select the subset with the proposed metric, we introduce Progressive Chunked Maximal Marginal Relevance. Extensive experiments demonstrate that our method achieves a better trade off between performance and efficiency, while providing another perspective on existing pruning approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。