通过注意力引导提升大模型语音合成稳定性，减少重复或缺失发音
Eliminating stability hallucinations in llm-based tts models via attention guidance
- 用维特比算法计算文本与语音对齐质量，提出最优对齐评分
- 在CosyVoice2训练中引入该评分，使语音生成更连续稳定
- 利用预训练注意力值引导学生模型，有效降低合成中的幻觉问题
本文针对基于大模型的文本转语音（TTS）系统中存在的稳定性幻觉（如重复或遗漏语音）问题，通过改进和利用注意力机制进行解决。首先分析了大模型中文本标记与语音标记之间的对齐机制，提出一种名为最优对齐评分（OAS）的新指标，采用维特比算法评估文本-语音对齐质量。随后，将OAS融入CosyVoice2的训练过程，帮助大模型学习连续稳定的对齐关系。此外，还利用预训练注意力值通过思维链（CoT）方式指导学生模型训练，进一步减少合成语音中的稳定性幻觉。在Seed-TTS-Eval和CV3-Eval两个测试集上的实验表明，所提方法能有效降低CosyVoice2的稳定性幻觉，且未引入额外负面影响。
原文摘要 · Abstract (English)
This paper focuses on resolving stability hallucinations (e.g., repetitive or omitted speech) in LLM-based Text-to-Speech (TTS) models by improving and leveraging the attention mechanism. First, we analyzed the alignment mechanism between text tokens and speech tokens in LLMs. We then proposed a metric termed the Optimal Alignment Score (OAS), which employs the Viterbi algorithm to evaluate text-speech alignment quality. Subsequently, OAS was integrated into the training of CosyVoice2 to assist LLMs in learning continuous, stable alignment. Additionally, the pre-trained attention value is employed to guide the training of the student CosyVoice2 via chain-of-thought (CoT), which further reduces stability hallucinations in synthesized speech. Experiments on the Seed-TTS-Eval and CV3-Eval test sets demonstrate that the proposed methods can effectively reduce the stability hallucinations of CosyVoice2 without introducing additional negative effects. The appendix is available at https://wsmzzz.github.io/llm_attn.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。