用翻译任务增强语音编码器预训练，提升语音大模型表现
Does Translation-Enhanced Speech Encoder Pre-training Affect Speech LLMs?
- 在预训练中加入语音翻译目标，使编码器输出更适配大语言模型的通用空间
- 跨模态融合能力显著提升，多个下游语音大模型任务性能更优
- 适合研究语音与语言模型融合、追求高性能语音大模型的开发者
将预训练语音编码器与大型语言模型（LLM）连接是构建语音大模型的标准架构。然而，编码器与LLM之间存在结构错位：不同于基于自动语音识别的编码器常生成语言特异的表示，LLM运行于统一的语言无关空间。需建立机制将编码器的语言特异性表示对齐至共享空间。我们提出，语音翻译可提供一种合理解决方案——相比单语转录，翻译要求模型跨越不同语言，学习语言无关表征。我们实验评估了在语音编码器预训练中引入翻译目标的影响。结果表明，翻译增强的预训练提升了跨模态融合能力，并在多个下游语音大模型任务中取得更优性能。
原文摘要 · Abstract (English)
Connecting a pre-trained speech encoder to a Large Language Model (LLM) is the standard architecture for building Speech LLMs. However, a structural misalignment exists between the encoder and the LLM. Unlike encoders based on automatic speech recognition, which often produce representations in separate language-specific spaces, LLMs operate within a unified language-agnostic space. A mechanism is required to align the encoder's language-specific representations with the LLM's shared space. We argue that speech translation provides a principled way to achieve this. Unlike monolingual transcription, translation requires the model to bridge different languages and learn language-agnostic representations. We experimentally evaluate the impact of incorporating translation objectives into speech encoder pre-training. Our results demonstrate that translation-enhanced pre-training improves cross-modal integration and leads to superior performance across downstream Speech LLM tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。