通过音视频交互机制动态剪枝，提升多模态生成效率
Audio-Visual Exchange-Aware Token Pruning for Efficient Audio-Visual Captioning
- 基于强化学习设计音视频跨模态令牌交换策略
- 在40%保留率下保持原模型性能（VILA 1.5-8B: 54.5 vs 54.6）
- 适合追求高效多模态生成的开发者与研究者
音视频字幕生成旨在从视频和音频内容中生成自然语言描述。多模态大模型虽已推动该任务发展，但双模态输入会向大模型注入大量令牌，导致预填充自注意力计算复杂度呈二次增长。现有令牌剪枝方法通常依赖注意力、显著性或交叉熵损失进行保留，但硬阈值选择难以保留真正有价值的高混淆度令牌（尤其在决策边界附近）。为此，本文提出一种基于强化学习的音视频动态令牌剪枝方法——AVEX-Prune。其核心是引入音视频令牌交换机制：通过替换低置信度保留令牌为同模态或异模态中的高置信度候选令牌，并评估令牌互换对字幕生成的影响，从而筛选出真正有价值的令牌。实验表明，该方法在保持40%令牌保留率的前提下，于VILA 1.5-8B（54.5 vs. 54.6）与VideoLLaMA 2（57.0 vs. 56.8）上均维持了全令牌质量。
原文摘要 · Abstract (English)
Audio-visual captioning generates natural language descriptions from video and audio content. Multimodal LLMs have advanced this task, but both modalities contribute many tokens to the LLM input, where prefill self-attention scales quadratically. Existing token-pruning methods usually retain tokens by attention, saliency, or cross-entropy loss, yet the hard threshold selection makes it difficult to retain tokens that are truly valuable, especially for high-confusing tokens near the decision boundary. To this end, we propose a AVEX-Prune, an RL-based audio-visual dynamic token pruning method in this work. In our AVEX-Prune, an audio-visual token exchange strategy is proposed to select truly valuable tokens by replacing low-confidence retained tokens with high-confidence candidate tokens from the same or the other modality, and measuring the differences in caption generation from token swaps. AVEX-Prune preserves full-token quality at a 40% retention ratio on both VILA 1.5-8B (54.5 vs. 54.6) and VideoLLaMA 2 (57.0 vs. 56.8).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。