用合成语音训练语音识别,只需真实数据的1/4就能超越全真实数据模型。
How to Leverage Synthetic Speech for LLM-Based ASR Systems?

- 通过分析大语言模型中间层识别合成语音的线索,定位到早期至中期层最易区分真伪。
- 将合成语音与房间脉冲响应混合,可还原真实录音的声学不规则性,缩小分布差距。
- 结合层选择模块和混响增强,在仅用13.6小时真实语音时性能超全真实数据基线。
在银行、医疗等受监管领域,因隐私限制,真实语音数据收集与留存成本高昂,现代文本转语音(TTS)生成的合成语音成为训练自动语音识别(ASR)系统的替代方案。然而,合成与真实数据间持续存在的分布差异限制了其替代程度。以往研究多将此差距视为黑箱处理,而本文直接探究其根源,通过分析SLAM-ASR架构发现:其大语言模型(LLM)主干在早期至中期层对真实与合成语音具有最强区分能力,且时间与语调扰动在此处最易破坏判别信号。进一步表明,表示层可分性虽有帮助,但无法直接预测下游ASR性能提升。相反,将合成语音与房间脉冲响应(RIRs)卷积,能有效缩小差距,非因声音更清晰或自然,而是重现真实录音中的声学不规则性。基于此,设计层选择模块并结合RIR增强,在仅使用25%真实语音(13.6小时)时即匹配全真实数据基线,并在更高比例下表现更优。
原文摘要 · Abstract (English)
In regulated domains such as banking and healthcare, where privacy constraints make real speech costly to collect and retain, synthetic speech from modern text-to-speech (TTS) is an appealing alternative for training automatic speech recognition (ASR) without exposing sensitive customer recordings. Yet a persistent distributional gap between synthetic and real data limits how far it can replace genuine recordings. Prior work largely treats this gap as a black box to be engineered around, but in our work, we instead examine its origin directly by probing a SLAM-ASR architecture. Then, we localise where its LLM backbone separates real from synthetic speech and find the discriminative signal concentrated in the early-to-middle layers, where temporal and prosodic perturbations disrupt it most. We further show that representation-level separability, help, but does not directly predict downstream ASR gains. On the other hand, convolving synthetic audio with room impulse responses (RIRs) narrows the gap not by making synthetic speech sound cleaner or more natural, but by reproducing the acoustic irregularities of real recordings. Translating these findings into the training procedure, by adding a layer-selection module combined with RIR augmentation matches a fully real-data baseline using only 25% of the real speech (13.6h) and surpasses it at all higher proportions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。