改进Whisper模型，让其更好识别混合语言语音。
Adapting Whisper for Code-Switching through Encoding Refining and Language-Aware Decoding
- 用编码器精炼模块增强句内语言切换识别能力。
- 引入语言感知解码器，使模型在不同层区分语言。
- 在非母语语言上表现显著提升，适合多语言语音场景。
混合语言自动语音识别（CS-ASR）因口音、听觉相似性和无缝语言切换导致语言混淆而面临挑战。本文将大规模多语言预训练模型Whisper应用于混合语言场景，从编码器和解码器两方面进行改进。首先提出编码器精炼模块，提升编码器对句内语言切换的建模能力；其次设计两组语言感知适配器，结合不同语言提示嵌入，在解码器每一层实现语言特异性信息，并通过融合模块整合解码结果。在SEAME数据集上的实验表明，相较于基线模型，所提方法在dev_man和dev_sge测试集上分别实现4.1%和7.2%的相对误识别率（MER）降低，优于现有最先进方法。实验还发现该方法显著提升非母语语言的识别性能，表明其能更有效区分双语语音中的语言边界。
原文摘要 · Abstract (English)
Code-switching (CS) automatic speech recognition (ASR) faces challenges due to the language confusion resulting from accents, auditory similarity, and seamless language switches. Adaptation on the pre-trained multi-lingual model has shown promising performance for CS-ASR. In this paper, we adapt Whisper, which is a large-scale multilingual pre-trained speech recognition model, to CS from both encoder and decoder parts. First, we propose an encoder refiner to enhance the encoder's capacity of intra-sentence swithching. Second, we propose using two sets of language-aware adapters with different language prompt embeddings to achieve language-specific decoding information in each decoder layer. Then, a fusion module is added to fuse the language-aware decoding. The experimental results using the SEAME dataset show that, compared with the baseline model, the proposed approach achieves a relative MER reduction of 4.1% and 7.2% on the dev_man and dev_sge test sets, respectively, surpassing state-of-the-art methods. Through experiments, we found that the proposed method significantly improves the performance on non-native language in CS speech, indicating that our approach enables Whisper to better distinguish between the two languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。