多语言语音识别引入对话上下文,提升跨语言识别准确率。
Speak in Context: Multilingual ASR with Speech Context Alignment via Contrastive Learning

- 用轻量投影模块连接预训练语音与语言模型,支持多语言输入。
- 对比学习使语音与上下文表征对齐,在11种语言上提升超5%准确率。
- 适合需要处理多语种对话场景的语音系统研发者使用。
自动语音识别(ASR)得益于预训练语音与语言模型的发展,但多数系统仍局限于单语言环境和孤立短语。尽管近期上下文感知的ASR展现潜力,仍面临两大挑战:多语言支持有限,且语音与上下文表征缺乏合理对齐。本文提出一种支持多语言、多口音的上下文感知多语言ASR框架,结合冻结的语音编码器与仅解码器语言模型,通过轻量级投影模块实现结构化上下文提示(如对话历史、偏置词)引导转录。为增强语音与上下文间的交互,采用对比学习目标,在共享嵌入空间中对齐二者表征。在覆盖11种语言和5种英语方言、总计超过1500小时的真实对话语音数据集上评估表明,引入上下文可持续提升识别质量;对比对齐在不同上下文类型下带来额外增益,整体性能提升超过5%。结果凸显了上下文建模与跨模态对齐在多语言ASR中的重要性。
原文摘要 · Abstract (English)
Automatic speech recognition (ASR) has benefited from advances in pretrained speech and language models, yet most systems remain constrained to monolingual settings and short, isolated utterances. While recent efforts in context-aware ASR show promise, two key challenges persist: limited multilingual support and the absence of principled alignment between speech and contextual representations. In this paper, we introduce a context-aware multilingual ASR framework that supports diverse languages and accents while preserving the modularity of pretrained models. Our approach combines a frozen speech encoder and a decoder-only language model via a lightweight projection module, allowing structured context prompts, including dialogue history and biasing words, to guide transcription. To improve interaction between speech and context, we employ a contrastive learning objective that aligns their representations in a shared embedding space. Evaluations on over 1,500 hours of real-world conversational speech across 11 languages and 5 English dialects show that contextual input consistently improves recognition quality. Contrastive alignment provides additional gains when applied to different context types, with an overall performance gain of over 5%. These results highlight the importance of both contextual modeling and cross-modal alignment in multilingual ASR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。