压缩音频令牌，让大模型更快更省资源。
Towards Audio Token Compression in Large Audio Language Models
- 用无监督分段和平均池化减少音频令牌数量
- 压缩后性能接近原始帧级模型，令牌数减少三倍
- 适合需要高效处理音频的大模型应用
大型音频语言模型（LALMs）在语音与音频任务中表现优异，但其音频编码器生成的令牌序列速率较高（如25个令牌/秒），导致注意力计算开销大，制约可扩展性。本文探索了无监督分段、均匀平均池化等技术，在音频令牌输入LLM解码器前进行压缩。为缓解性能下降，微调时引入低秩适配器。在自动语音识别与语音到语音翻译任务上评估，这两项任务依赖于有效揭示输入信号的底层词汇内容。实验表明，压缩后的LALMs在将输入音频令牌数减少至原来的1/3时，仍能逼近帧级模型的性能。
原文摘要 · Abstract (English)
Large Audio Language Models (LALMs) deliver strong performance across speech and audio tasks, but their audio encoders generate high-rate token sequences (e.g., 25 tokens/s), making attention computation costly and limiting scalability. In this paper, we explore techniques such as unsupervised segmentation, uniform average pooling, etc., to reduce the number of audio tokens before they are consumed by the LLM decoder. To mitigate potential performance degradation, we employ low-rank adapters during finetuning. We evaluate our proposed models on two tasks, automatic speech recognition and speech-to-speech translation tasks, that are dependent on effectively uncovering the underlying lexical content of the input signal, and study the effect of downsampling on these tasks. Experimental results show that compressed LALMs can achieve performance closer to frame-level LALMs while reducing the input audio token count up to three times before the LLM backbone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。