端到端同步重建失语语音，响应快且更清晰。
End-to-End Simultaneous Dysarthric Speech Reconstruction with Frame-Level Adaptor and Multiple Wait-k Knowledge Distillation
- 用帧级适配模块融合声学与语义信息，提升容错性。
- 多等待k知识蒸馏让语音韵律更自然，词错误率降54.25%。
- 适合失语症患者实时沟通，尤其对严重发音障碍者有效。
失语语音重建（DSR）通常采用级联系统，结合自动语音识别（ASR）与句级文本转语音（TTS），将失语语音转换为正常语调语音。然而，失语患者常说话较慢，导致此类系统响应过长，在长语音场景下不实用。基于流式ASR和增量TTS的级联系统可降低延迟，但不同严重程度的失语患者对同一文本发音差异大，影响ASR鲁棒性，限制重建语音可懂度；此外，增量TTS因感受野有限，韵律预测效果差。本文提出一种端到端同步DSR系统，包含两项创新：1）引入帧级适配模块，通过显式-隐式语义信息融合与联合训练，增强TTS对ASR输出的容错能力；2）设计多等待k自回归TTS模块，利用多视角知识蒸馏缓解韵律退化。系统在Tesla A100上平均响应时间为1.03秒，平均实时因子（RTF）为0.71。在UASpeech数据集上，平均主观评分（MOS）达4.67，相比现有最优方法词错误率（WER）相对降低54.25%。演示地址：https://wflrz123.github.io/
原文摘要 · Abstract (English)
Dysarthric speech reconstruction (DSR) typically employs a cascaded system that combines automatic speech recognition (ASR) and sentence-level text-to-speech (TTS) to convert dysarthric speech into normally-prosodied speech. However, dysarthric individuals often speak more slowly, leading to excessively long response times in such systems, rendering them impractical in long-speech scenarios. Cascaded DSR systems based on streaming ASR and incremental TTS can help reduce latency. However, patients with differing dysarthria severity exhibit substantial pronunciation variability for the same text, resulting in poor robustness of ASR and limiting the intelligibility of reconstructed speech. In addition, incremental TTS suffers from poor prosodic feature prediction due to a limited receptive field. In this study, we propose an end-to-end simultaneous DSR system with two key innovations: 1) A frame-level adaptor module is introduced to bridge ASR and TTS. By employing explicit-implicit semantic information fusion and joint module training, it enhances the error tolerance of TTS to ASR outputs. 2) A multiple wait-k autoregressive TTS module is designed to mitigate prosodic degradation via multi-view knowledge distillation. Our system has an average response time of 1.03 seconds on Tesla A100, with an average real-time factor (RTF) of 0.71. On the UASpeech dataset, it attains a mean opinion score (MOS) of 4.67 and demonstrates a 54.25% relative reduction in word error rate (WER) compared to the state-of-the-art. Our demo is available at: https://wflrz123.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。