用轻量级编码器混合提升音频大模型多任务能力
MoWE-Audio: Multitask AudioLLMs with Mixture of Weak Encoders
- 引入弱编码器混合机制，动态选择轻量编码器增强特征提取
- 在多个音频任务上实现性能提升，且模型规模几乎不变
- 适合需要高效适配新音频任务的研究者和开发者
大语言模型（LLM）的快速发展显著提升了自然语言处理能力，推动了能够同时处理语音与音频输入的AudioLLM的发展。现有AudioLLM通常将预训练音频编码器与预训练LLM结合，并在特定音频任务上微调。然而，预训练音频编码器在捕捉新任务与数据集特征方面能力有限。为此，我们提出在AudioLLM框架中引入弱编码器混合（MoWE），通过一个基础编码器搭配一组轻量级编码器池，在输入音频驱动下按需激活，以增强特征提取而几乎不增加模型规模。实证结果表明，MoWE能有效提升多任务性能，扩大AudioLLM在多样化音频任务中的适用性。
原文摘要 · Abstract (English)
The rapid advancements in large language models (LLMs) have significantly enhanced natural language processing capabilities, facilitating the development of AudioLLMs that process and understand speech and audio inputs alongside text. Existing AudioLLMs typically combine a pre-trained audio encoder with a pre-trained LLM, which are subsequently finetuned on specific audio tasks. However, the pre-trained audio encoder has constrained capacity to capture features for new tasks and datasets. To address this, we propose to incorporate mixtures of `weak' encoders (MoWE) into the AudioLLM framework. MoWE supplements a base encoder with a pool of relatively light weight encoders, selectively activated based on the audio input to enhance feature extraction without significantly increasing model size. Our empirical results demonstrate that MoWE effectively improves multi-task performance, broadening the applicability of AudioLLMs to more diverse audio tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。