让语音大模型根据任务自动选择最佳音频特征，提升多任务表现
Enhancing Speech Large Language Models with Prompt-Aware Mixture of Audio Encoders
- 按任务提示动态选择不同音频编码器提取特征
- 单一模型在语音识别、说话人数量验证等任务上超越单编码器最优表现
- 适合需要多音频理解任务的语音大模型研究者
将音频编码器与大语言模型（LLM）结合，使LLM能够完成自动语音识别（ASR）和音频描述（AC）等音频理解任务。现有研究多通过适配层生成统一音频特征，但不同任务对语义或声学特征的需求不同，因此任务特定特征更优。本文提出提示感知混合（PaM），利用多个音频编码器，根据提示信息动态选择最合适的特征。实验表明，仅使用一个语音大模型配合PaM，即可在ASR、说话人数量验证和音频描述任务上超越所有单编码器语音大模型的最佳表现。PaM还优于拼接、平均等特征融合基线方法。
原文摘要 · Abstract (English)
Connecting audio encoders with large language models (LLMs) allows the LLM to perform various audio understanding tasks, such as automatic speech recognition (ASR) and audio captioning (AC). Most research focuses on training an adapter layer to generate a unified audio feature for the LLM. However, different tasks may require distinct features that emphasize either semantic or acoustic aspects, making task-specific audio features more desirable. In this paper, we propose Prompt-aware Mixture (PaM) to enhance the Speech LLM that uses multiple audio encoders. Our approach involves using different experts to extract different features based on the prompt that indicates different tasks. Experiments demonstrate that with PaM, only one Speech LLM surpasses the best performances achieved by all single-encoder Speech LLMs on ASR, Speaker Number Verification, and AC tasks. PaM also outperforms other feature fusion baselines, such as concatenation and averaging. Our code would be available at: https://github.com/shanweiqiao/PaM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。