融合多个语音编码器，提升大模型语音识别性能。
Speech Encoder Fusion for LLM-based Automatic Speech Recognition
- 用可学习的融合策略整合多个预训练语音编码器。
- 多语言与说话人分离场景下均实现性能提升。
- 适合追求高精度语音识别的开发者使用。
具备语音感知能力的大语言模型可通过预训练的声学编码器将语音特征映射到语言模型嵌入空间。尽管语音编码器的选择对性能影响显著，但不同编码器常表现出互补优势，这促使我们探索其组合的可能性。本文研究了融合多个预训练语音编码器是否能增强基于大语言模型的自动语音识别（ASR）系统。我们提出了超越简单特征拼接的多种融合策略，包括可学习的组合方式及基于Transformer的融合架构，并在单语、多语及说话人分离的语音识别场景中进行了评估。结果表明，在所有测试场景下，精心设计的并行语音编码器融合均能有效提升下游性能，且计算开销有限。
原文摘要 · Abstract (English)
Speech-aware large language models (LLMs) can incorporate speech through pre-trained acoustic encoders that project speech features into the LLM embedding space. While the choice of the speech encoder critically influences performance, different encoders often exhibit complementary strengths, motivating their combination. In this work, we investigate whether fusing multiple pre-trained speech encoders can enhance speech-aware LLMs for automatic speech recognition (ASR). We explore several fusion strategies beyond simple feature concatenation, including learned combinations and Transformer-based fusion architectures, and evaluate them across mono- and multilingual ASR settings as well as diarized speech recognition. Our results indicate that carefully fusing multiple parallel speech encoders improves downstream performance in all scenarios with limited computational overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。