动态分配注意力头权重，实现音频令牌高效压缩
HeadRouter: Dynamic Head-Weight Routing for Task-Adaptive Audio Token Pruning in Large Audio Language Models
- 根据任务动态调整注意力头重要性，精准保留关键音频信息
- 在保留70%令牌时，模型性能仍超基线101.8%~103.0%
- 无需训练，适配多种大模型，尤其适合长音频处理场景
近期的大规模音频语言模型（LALMs）在处理长时多模态序列方面表现出色，但推理成本高昂。令牌压缩可直接减少序列中的冗余信息。现有方法通常假设所有注意力头对各类音频任务贡献均等，通过平均各头得分来计算令牌重要性。然而我们的分析表明，注意力头在不同音频领域表现差异显著；仅有少量头对音频响应活跃，且在语义与声学任务中表现迥异。为此，我们提出HeadRouter，一种基于头重要性的训练无关令牌剪枝方法，能感知不同任务下注意力头的差异化贡献，最大化关键令牌保留。在AudioMarathon和MMAU-Pro基准测试中，该方法实现业界最优压缩性能：即使保留70%音频令牌，仍在Qwen2.5-Omni-3B和Qwen2.5-Omni-7B上分别达到基线101.8%和103.0%的平均性能。
原文摘要 · Abstract (English)
Recent large audio language models (LALMs) demonstrate remarkable capabilities in processing extended multi-modal sequences, yet incur high inference costs. Token compression is an effective method that directly reduces redundant tokens in the sequence. Existing compression methods usually assume that all attention heads in LALMs contribute equally to various audio tasks and calculate token importance by averaging scores across all heads. However, our analysis demonstrates that attention heads exhibit distinct behaviors across diverse audio domains. We further reveal that only a sparse subset of attention heads actively responds to audio, with completely different performance when handling semantic and acoustic tasks. In light of this observation, we propose HeadRouter, a head-importance-aware token pruning method that perceives the varying importance of attention heads in different audio tasks to maximize the retention of crucial tokens. HeadRouter is training-free and can be applied to various LALMs. Extensive experiments on the AudioMarathon and MMAU-Pro benchmarks demonstrate that HeadRouter achieves state-of-the-art compression performance, exceeding the baseline model even when retaining 70% of the audio tokens and achieving 101.8% and 103.0% of the vanilla average on Qwen2.5-Omni-3B and Qwen2.5-Omni-7B, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。