提出首个面向音视频大模型的高效令牌剪枝框架，显著降低计算量。
FastAV: Efficient Token Pruning for Audio-Visual Large Language Model Inference
- 基于注意力权重分阶段剪枝，先全局后精细
- 在两个主流音视频模型上减少超40%计算量
- 兼容FlashAttention等高效注意力机制
本文提出FastAV，首个专为音视频大语言模型（AV-LLMs）设计的令牌剪枝框架。尽管令牌剪枝在标准大语言模型（LLMs）和视觉-语言模型（LVLMs）中已受关注，但其在需处理多模态数据的AV-LLMs中的应用仍鲜有研究。为此，我们提出一种基于注意力权重分析的剪枝策略，识别不同阶段的关键令牌，并实施两阶段剪枝：（1）在中间层进行全局剪枝，移除普遍影响较小的令牌；（2）在后期层结合对下一令牌生成的影响进行精细化剪枝。值得注意的是，该方法无需完整注意力图，完全兼容FlashAttention等高效注意力机制。大量实验表明，FastAV在两个代表性AV-LLMs上实现超过40%的浮点运算量（FLOPs）减少，同时保持或提升模型性能。
原文摘要 · Abstract (English)
In this work, we present FastAV, the first token pruning framework tailored for audio-visual large language models (AV-LLMs). While token pruning has been actively explored in standard large language models (LLMs) and vision-language models (LVLMs), its application to AV-LLMs has received little attention, even though multimodal integration substantially increases their token demands. To address this gap, we introduce a pruning strategy that utilizes attention weights to identify tokens emphasized at different stages and estimates their importance. Building on this analysis, FastAV applies a two-stage pruning strategy: (1) global pruning in intermediate layers to remove broadly less influential tokens, and (2) fine pruning in later layers considering the impact on next token generation. Notably, our method does not rely on full attention maps, which makes it fully compatible with efficient attention mechanisms such as FlashAttention. Extensive experiments demonstrate that FastAV reduces FLOPs by more than 40% on two representative AV-LLMs, while preserving or even improving model performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。