探究语音大模型与语言大模型连接方式，发现核心影响因素
How to Connect Speech Foundation Models and Large Language Models? What Matters and What Does Not
- 用适配模块将语音模型输出映射到语言模型空间
- 语音模型对任务表现影响最大，适配器影响中等且依赖组合
- 结果对选择适配器和模型组合有指导意义
大型语言模型（LLM）的卓越表现推动了其在多种任务和输入模态中的应用。在语音转文本（S2T）任务中，新方法是通过适配模块将语音基础模型（SFM）编码器的输出投影到LLM的嵌入空间。然而，尚无研究探讨下游任务性能对各组件（SFM、适配器、LLM）的依赖程度，也未考察最优适配器设计是否依赖所选的SFM和LLM。为填补这一空白，我们评估了5种适配模块、2个LLM（Mistral和Llama）以及2个SFM（Whisper和SeamlessM4T）在两个广泛使用的S2T任务——自动语音识别和语音翻译上的组合表现。结果表明，SFM在下游性能中起决定性作用，而适配器的选择影响中等，并依赖于SFM和LLM的组合。
原文摘要 · Abstract (English)
The remarkable performance achieved by Large Language Models (LLM) has driven research efforts to leverage them for a wide range of tasks and input modalities. In speech-to-text (S2T) tasks, the emerging solution consists of projecting the output of the encoder of a Speech Foundational Model (SFM) into the LLM embedding space through an adapter module. However, no work has yet investigated how much the downstream-task performance depends on each component (SFM, adapter, LLM) nor whether the best design of the adapter depends on the chosen SFM and LLM. To fill this gap, we evaluate the combination of 5 adapter modules, 2 LLMs (Mistral and Llama), and 2 SFMs (Whisper and SeamlessM4T) on two widespread S2T tasks, namely Automatic Speech Recognition and Speech Translation. Our results demonstrate that the SFM plays a pivotal role in downstream performance, while the adapter choice has moderate impact and depends on the SFM and LLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。