一个模型同时搞定语音识别和语音合成，省一半参数还保持高性能。
STTATTS: Unified Speech-To-Text And Text-To-Speech Model
- 用多任务学习共享参数，统一训练语音识别与语音合成。
- 性能接近独立模型，参数量减少约50%。
- 适合资源匮乏语言，代码模型开源可用。
语音识别（ASR）与语音合成（TTS）模型通常分别训练，各自拥有独立的学习目标、数据和参数，导致两个大型网络并存。本文提出一种参数高效的联合学习方法，通过多任务学习目标和共享参数实现ASR与TTS的统一建模。实验表明，该多任务模型性能与独立训练模型相当，但总参数量减少约50%。我们在英语（资源丰富）和阿拉伯语（语音合成数据较少）上进行了测试，所有模型均使用公开数据训练，训练代码与模型检查点均已开源，便于后续研究。
原文摘要 · Abstract (English)
Speech recognition and speech synthesis models are typically trained separately, each with its own set of learning objectives, training data, and model parameters, resulting in two distinct large networks. We propose a parameter-efficient approach to learning ASR and TTS jointly via a multi-task learning objective and shared parameters. Our evaluation demonstrates that the performance of our multi-task model is comparable to that of individually trained models while significantly saving computational and memory costs ($\sim$50\% reduction in the total number of parameters required for the two tasks combined). We experiment with English as a resource-rich language, and Arabic as a relatively low-resource language due to shortage of TTS data. Our models are trained with publicly available data, and both the training code and model checkpoints are openly available for further research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。