动态调整视觉令牌压缩率，提升多模态模型生成效率。
Dynamic Token Reduction during Generation for Vision Language Models
- 根据注意力分布动态调整视觉令牌压缩率。
- 生成过程后期压缩率更高，减少计算开销30%以上。
- 适合需要高效推理的多模态应用开发人员。
视觉语言模型（VLMs）在多模态任务中表现优异，但受限于解码器注意力机制的二次复杂度和自回归生成过程。现有方法如FASTV和VTW虽能减少冗余视觉令牌，但仅在单次前向传播中进行剪枝，未系统分析生成全过程中的令牌冗余性。本文提出一种针对VLMs的动态剪枝策略——DyRate，可随生成过程逐步调整压缩率。我们对注意力分布的分析表明，视觉令牌的重要性在生成过程中逐渐降低，因此可采用更激进的压缩策略。通过引入基于注意力分布的轻量级预测器，实现剪枝率的灵活调整。实验结果表明，该方法在显著降低计算需求的同时，保持了响应质量。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) have achieved notable success in multimodal tasks but face practical limitations due to the quadratic complexity of decoder attention mechanisms and autoregressive generation. Existing methods like FASTV and VTW have achieved notable results in reducing redundant visual tokens, but these approaches focus on pruning tokens in a single forward pass without systematically analyzing the redundancy of visual tokens throughout the entire generation process. In this paper, we introduce a dynamic pruning strategy tailored for VLMs, namedDynamic Rate (DyRate), which progressively adjusts the compression rate during generation. Our analysis of the distribution of attention reveals that the importance of visual tokens decreases throughout the generation process, inspiring us to adopt a more aggressive compression rate. By integrating a lightweight predictor based on attention distribution, our approach enables flexible adjustment of pruning rates based on the attention distribution. Our experimental results demonstrate that our method not only reduces computational demands but also maintains the quality of responses.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。