用纯文本训练语音识别模型，显著降低错误率。
WhisTLE: Deeply Supervised, Text-Only Domain Adaptation for Pretrained Speech Recognition Transformers
- 通过变分自编码器学习文本到隐向量的映射，实现文本驱动的模型适配。
- 结合TTS增强后，词错误率相对降低49.0%，112种场景中100次表现最优。
- 可无缝融入现有流程，适合需快速适配新领域的语音识别应用。
预训练语音识别模型如Whisper在未见语域下仍需领域适配。在真实场景中收集语音数据常不现实，因此需依赖纯文本适配。本文提出WhisTLE，一种针对预训练编码器-解码器语音识别模型的深度监督、纯文本领域适配方法。WhisTLE训练一个变分自编码器（VAE）以建模从文本生成的编码器输出，并使用学习到的文本到隐向量编码器微调解码器，可选地结合文本到语音（TTS）适配。推理时恢复原始编码器，无额外运行时开销。在四个数据集和四种ASR模型上，结合TTS的WhisTLE将词错误率（WER）相对降低49.0%，在112种场景中的100种优于所有非WhisTLE基线。我们还发现WhisTLE可与任何其他领域适配方法叠加互补；因此建议在标准流程中纳入WhisTLE以适配编码器-解码器型语音识别模型。
原文摘要 · Abstract (English)
Pretrained automatic speech recognition (ASR) models such as Whisper perform well but still need domain adaptation to handle unseen parlance. In many real-world settings, collecting speech data is impractical, necessitating text-only adaptation. We propose WhisTLE, a deeply supervised, text-only adaptation method for pretrained encoder-decoder ASR models. WhisTLE trains a variational autoencoder (VAE) to model encoder outputs from text and fine-tunes the decoder using the learned text-to-latent encoder, optionally combined with text-to-speech (TTS) adaptation. At inference, the original encoder is restored, incurring no extra runtime cost. Across four datasets and four ASR models, WhisTLE with TTS reduces word error rate (WER) by a relative 49.0% and outperforms all non-WhisTLE baselines in 100 of 112 scenarios. We also find that WhisTLE additively complements any combination of other domain adaptation approaches; we thus recommend the inclusion of WhisTLE during standard processes for adapting encoder-decoder ASR models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。