一个无需自回归的模型，能同时处理语音识别与合成。
A Non-autoregressive Model for Joint STT and TTS
- 非自回归架构，可联合处理语音与文本输入
- 在无配对数据下仍能训练，性能超越专用识别模型
- 通过迭代反馈优化，提升语音与文本生成质量
本文提出一种完全非自回归的联合建模框架，用于自动语音识别（STT）与语音合成（TTS）。该模型具备多模态能力，可单独或联合处理语音与文本输入。得益于其多模态特性，模型可在仅含未配对语音或文本数据的情况下进行训练。我们进一步设计了迭代精炼策略，将输出部分假设反馈至输入，逐步优化STT与TTS预测结果。实验表明，该联合模型在各项任务中均优于专用的STT基线模型，在多数评估指标上与专用的TTS基线模型表现相当。
原文摘要 · Abstract (English)
In this paper, we take a step towards jointly modeling automatic speech recognition (STT) and speech synthesis (TTS) in a fully non-autoregressive way. We develop a novel multimodal framework capable of handling the speech and text modalities as input either individually or together. The proposed model can also be trained with unpaired speech or text data owing to its multimodal nature. We further propose an iterative refinement strategy to improve the STT and TTS performance of our model such that the partial hypothesis at the output can be fed back to the input of our model, thus iteratively improving both STT and TTS predictions. We show that our joint model can effectively perform both STT and TTS tasks, outperforming the STT-specific baseline in all tasks and performing competitively with the TTS-specific baseline across a wide range of evaluation metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。