实现可实时生成的多说话人对话语音,支持自然换人与情感连贯。
FireRedTTS-2: Towards Long Conversational Speech Generation for Podcast and Chatbot
- 采用12.5Hz流式语音分词器,支持长对话实时生成
- 在播客场景中优于MoonCast等系统,换人准确率更高
- 适合聊天机器人与广播类实时语音应用
当前对话语音合成方法通常需完整对话文本才开始生成,且输出为不可分割的单一语音,难以用于交互式聊天;同时存在合成不稳定、说话人切换不准、语调不连贯等问题。本文提出FireRedTTS-2,一种面向多说话人的长对话流式语音合成系统,实现稳定自然的语音输出,具备可靠的说话人切换与上下文感知的语调表现。引入新型12.5Hz流式语音分词器,加速训练与推理,延长最大对话长度,增强语义编码能力,支持高保真流式生成。采用文本-语音交错格式,按时间顺序拼接带说话人标签的文本与对齐语音标记,并使用双变换器架构:大解码器在第一层预测标记,小模型完成后续层建模。实验表明,FireRedTTS-2可无缝集成至聊天框架,仅需少量微调即可生成受上下文隐含线索引导的情绪化语音。在播客生成任务中,其在客观清晰度、说话人转接可靠性及感知自然度方面均超越现有系统(包括MoonCast、Zipvoice-Dialogue和MOSS-TTSD),且语调与上下文一致。演示地址:https://fireredteam.github.io/demos/firered_tts_2。
原文摘要 · Abstract (English)
Current dialogue generation approaches typically require the complete dialogue text before synthesis and produce a single, inseparable speech containing all voices, making them unsuitable for interactive chat; moreover, they suffer from unstable synthesis, inaccurate speaker transitions, and incoherent prosody. In this work, we present FireRedTTS-2, a long-form streaming TTS system for multi-speaker dialogue generation, delivering stable, natural speech with reliable speaker switching and context-aware prosody. A new 12.5Hz streaming speech tokenizer accelerates training and inference, extends maximum dialogue length, encodes richer semantics to stabilize text-to-token modeling and supports high-fidelity streaming generation for real-time applications. We adopt a text-speech interleaved format, concatenating speaker-labeled text with aligned speech tokens in chronological order, and model it with a dual-transformer: a large decoder-only transformer predicts tokens at the first layer, and a smaller one completes subsequent layers. Experimental results show that FireRedTTS-2 integrates seamlessly with chat frameworks and, with minimal fine-tuning, produces emotionally expressive speech guided by implicit contextual cues. In podcast generation, it surpasses existing systems including MoonCast, Zipvoice-Dialogue, and MOSS-TTSD in objective intelligibility, speaker-turn reliability, and perceived naturalness with context-consistent prosody. Our demos are available at https://fireredteam.github.io/demos/firered_tts_2.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。