通过语言嵌入建模，提升多语言语音识别对未见语言的性能
Enhancing Multilingual ASR for Unseen Languages via Language Embedding Modeling
- 用语言标签嵌入加权和模拟未知语言特征
- 在零样本和微调设置下显著提升识别准确率
- 适合需要跨语言泛化的语音识别研究者
多语言自动语音识别（ASR）旨在单个系统中识别和转录多种语言的语音。Whisper 是当前最先进的多语言 ASR 模型，能有效处理 99 种语言，依靠海量数据并使用语言标签作为前缀引导识别过程。然而，尽管表现优异，Whisper 在未包含在其预训练中的未知语言上仍表现不佳。受许多语言具有相似语言特征的启发，我们提出方法利用这些共性来提升未知语言的识别性能。具体而言，我们引入一种加权求和方法，基于 Whisper 预测的语言概率，计算语言标签嵌入的加权和；此外，还设计了一种预测器方法，进一步优化该加权嵌入，使其更接近真实未知语言的嵌入。实验结果表明，在零样本和微调设置下，性能均有显著提升。所提方法优于基线模型，为解决多语言 ASR 中未见语言问题提供了有效方案。
原文摘要 · Abstract (English)
Multilingual Automatic Speech Recognition (ASR) aims to recognize and transcribe speech from multiple languages within a single system. Whisper, one of the most advanced ASR models, excels in this domain by handling 99 languages effectively, leveraging a vast amount of data and incorporating language tags as prefixes to guide the recognition process. However, despite its success, Whisper struggles with unseen languages, those not included in its pre-training. Motivated by the observation that many languages share linguistic characteristics, we propose methods that exploit these relationships to enhance ASR performance on unseen languages. Specifically, we introduce a weighted sum method, which computes a weighted sum of the embeddings of language tags, using Whisper's predicted language probabilities. In addition, we develop a predictor-based approach that refines the weighted sum embedding to more closely approximate the true embedding for unseen languages. Experimental results demonstrate substantial improvements in ASR performance, both in zero-shot and fine-tuning settings. Our proposed methods outperform baseline approaches, providing an effective solution for addressing unseen languages in multilingual ASR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。