用离散语义标记构建语音链，提升语音识别与合成效果
TokenChain: A Discrete Speech Chain via Semantic Token Modeling
- 通过语义标记耦合语音识别与合成，实现端到端反馈
- 在LibriSpeech上提前2-6轮达到更高准确率，错误率降低5-13%
- 在TED-LIUM上识别与合成错误率分别下降56%和31%，遗忘极少
机器语音链模拟人类感知-生成循环，在联合提升语音识别（ASR）与语音合成（TTS）方面表现有效。我们提出TokenChain，一种完全离散的语音链，将语义标记型ASR与两阶段TTS结合：一个与ASR共同训练的自回归文本到语义模型，以及仅用于合成的掩码生成式语义到声学模型。通过直通argmax/Gumbel-Softmax实现跨文本接口的端到端反馈，并利用动态权重平均平衡监督式ASR。消融实验研究了域内与跨域迁移的最优温度调度。评估显示，TokenChain在LibriSpeech上比基线早2-6个训练周期达到更高准确率，等周期误差降低5-13%，且保持稳定T2S性能；在TED-LIUM上相对ASR WER降低56%，T2S WER降低31%，且遗忘极小，表明以标记为接口的链式学习仍具有效性。
原文摘要 · Abstract (English)
Machine Speech Chain, simulating the human perception-production loop, proves effective in jointly improving ASR and TTS. We propose TokenChain, a fully discrete speech chain coupling semantic-token ASR with a two-stage TTS: an autoregressive text-to-semantic model co-trained with ASR and a masked-generative semantic-to-acoustic model for synthesis only. End-to-end feedback across the text interface is enabled with straight-through argmax/Gumbel-Softmax and balanced with supervised ASR via dynamic weight averaging. Ablations examine optimal temperature schedules for in- and cross-domain transfer. Evaluation reveals TokenChain surpasses baseline accuracy 2-6 epochs earlier and yields 5-13% lower equal-epoch error with stable T2S on LibriSpeech, and reduces relative ASR WER by 56% and T2S WER by 31% on TED-LIUM with minimal forgetting, showing that chain learning remains effective with token interfaces and models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。