用网页文本预训练，让语音识别模型学会混语表达。
AsyncSwitch: Asynchronous Text-Speech Adaptation for Code-Switched ASR
- 用海量混语文本先训练解码器，再对齐语音数据微调。
- 在马来-英语混语上使错误率降低9.02%，同时提升单语表现。
- 适合做多语言语音识别、尤其是资源少的混语场景。
构建混语语音识别系统面临语言歧义和数据稀缺的挑战，而真实混语语音采集成本高。现有方法通过文本生成合成语音，但计算开销大且难扩展。本文提出AsyncSwitch，一种异步适应框架：利用大规模文本丰富的网络数据，在少量配对语音-文本数据微调前，预先让ASR模型接触多样化的混语语境。该方法分三阶段：(1) 在混语文本上训练解码器的自注意力与前馈层；(2) 使用有限语音-文本数据通过交叉注意力对齐编码器与解码器；(3) 完整模型微调。在Whisper模型上针对马来-英语混语的实验显示，相对词错误率降低9.02%，同时提升了新加坡英语（Singlish）、马来语及其他英语变体的单语性能。
原文摘要 · Abstract (English)
Developing code-switched ASR systems is challenging due to language ambiguity and limited exposure to multilingual, code-switched data, while collecting such speech is costly. Prior work generates synthetic audio from text, but these methods are computationally intensive and hard to scale. We introduce AsyncSwitch, a novel asynchronous adaptation framework that leverages large-scale, text-rich web data to pre-expose ASR models to diverse code-switched domains before fine-tuning on paired speech-text corpora. Our three-stage process (1) trains decoder self-attention and feedforward layers on code-switched text, (2) aligns decoder and encoder via cross-attention using limited speech-text data, and (3) fully fine-tunes the entire model. Experiments with Whisper on Malay-English code-switching demonstrate a 9.02% relative WER reduction, while improving monolingual performance in Singlish, Malay, and other English variants.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。