仅用合成数据就能让语音识别模型适配新领域，无需真实语料。
A Domain Adaptation Framework for Speech Recognition Systems with Only Synthetic data
- 用大语言模型生成特定领域文本，再转为语音合成数据。
- 在音乐、天气、体育等域上降低10%~17%的错误率。
- 推理时只需额外9%实时因子，适合快速部署到多领域场景。
我们提出DAS（基于合成数据的领域自适应），一种针对预训练语音识别模型的新框架，可在无需真实数据的情况下高效适配各类语言定义的领域。DAS首先利用大语言模型生成特定领域的文本，再通过文本转语音技术转化为合成语音数据。这些数据用于微调Whisper模型，采用低秩适配器（LoRAs）实现对音乐、天气、体育等目标领域的精准适应。我们引入一种新型单次解码策略，在自回归生成过程中高效融合多个LoRA适配器的预测结果。实验表明，相比原模型，所有目标领域下的词错误率（WER）均降低10%至17%，且在非目标域上性能下降极小（如LibriSpeech测试集仅-1%）。此外，推理阶段仅增加9%的实时因子（RTF），使用三个LoRA适配器时仍具高效率。
原文摘要 · Abstract (English)
We introduce DAS (Domain Adaptation with Synthetic data), a novel domain adaptation framework for pre-trained ASR model, designed to efficiently adapt to various language-defined domains without requiring any real data. In particular, DAS first prompts large language models (LLMs) to generate domain-specific texts before converting these texts to speech via text-to-speech technology. The synthetic data is used to fine-tune Whisper with Low-Rank Adapters (LoRAs) for targeted domains such as music, weather, and sports. We introduce a novel one-pass decoding strategy that merges predictions from multiple LoRA adapters efficiently during the auto-regressive text generation process. Experimental results show significant improvements, reducing the Word Error Rate (WER) by 10% to 17% across all target domains compared to the original model, with minimal performance regression in out-of-domain settings (e.g., -1% on Librispeech test sets). We also demonstrate that DAS operates efficiently during inference, introducing an additional 9% increase in Real Time Factor (RTF) compared to the original model when inferring with three LoRA adapters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。