提出分段剪枝方法，显著降低音频-语言模型计算开销。
Segmentwise Pruning in Audio-Language Models
- 按时间维度设计轻量级分段剪枝策略，减少冗余音频标记。
- 仅保留四分之一初始标记时，Clotho v2上CIDEr下降不超过2%。
- 适合需处理长音频的场景，如语音理解与多模态任务。
近期音频-语言模型在多种音频任务中表现优异，且能处理长音频输入。然而其计算成本随序列长度急剧上升，而音频数据天然具有长序列特性。视觉-语言领域中，标记剪枝方法已证明可有效减少标记数量并保持性能。本文研究此类标记选择策略在音频-语言模型中的适用性，并提出一种考虑时间维度的轻量级改进方案。实验表明，在仅保留初始标记四分之一的情况下，Clotho v2数据集上CIDEr指标相对最大下降2%，MMAU数据集上准确率相对最大下降4%，验证了方法的有效性。
原文摘要 · Abstract (English)
Recent audio-language models have shown impressive performance across a wide range of audio tasks and are increasingly capable of handling long audio inputs. However, the computing costs in these models heavily depend on sequence length, which can become very large given the nature of audio data. In the vision-language domain, token pruning methods have proven effective in reducing token counts while preserving strong performance on standard benchmarks. In this work, we investigate the relevance and effectiveness of such token selection strategies in the context of audio-language models. We also improve them by proposing a lightweight strategy that takes the time dimension into account. While retaining only a quarter of the initial tokens, our approach results in a relative maximum decrease of 2% in CIDEr on Clotho v2 and a relative maximum decrease of 4% in accuracy on MMAU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。