arXiv:2608.17492cs.SD2026-08

用语义增强的语音表示,让语音生成更稳定可控。

FireRedTTS3: Unified Speech Generation and Editing with Semantically Enriched Speech Representations

论文配图:FireRedTTS3: Unified Speech Generation and Editing with Semantically Enriched Speech Representations
图 1 · 摘自论文原文
  • 用冻结的音频编码器做语义教师,规范语音特征空间
  • 零样本克隆和指令控制语音生成效果优于现有方法
  • 架构简单但能稳定生成高质量语音,适合语音编辑场景

近期连续自回归语音合成模型直接在连续语音表示上操作,保留丰富声学细节的同时利用文本大模型的指令理解能力。这一范式为语音克隆、指令控制语音设计和语音编辑开辟了新可能,但自回归生成中仍易积累误差。现有方案常需额外语义模块、多阶段分词器训练或复杂架构。本文提出FireRedTTS3,一种简洁有效的语音生成与编辑框架,从表示层面缓解误差累积。具体而言,采用在多样化语音理解任务上训练的冻结音频编码器作为语义教师,规范化音频特征空间,提升文本-语音对齐并稳定自回归生成,同时保持整体系统简洁。FireRedTTS3提供两个版本:FireRedTTS3-Base用于多语言、多方言零样本语音克隆;FireRedTTS3-Instruct实现统一的语音克隆、指令控制语音设计与语音编辑。实验表明,FireRedTTS3-Base在Seed-TTS-Eval和MiniMax-MLS-Test上平均语音可懂度与说话人相似度均最优;FireRedTTS3-Instruct在InstructTTSEval和Ming-Freeform-Audio-Edit上超越对比系统。结果证明,语义丰富的连续语音表示结合简洁架构,可实现稳定、可控、高保真的语音生成与编辑。代码与模型见https://github.com/FireRedTeam/FireRedTTS3。

原文摘要 · Abstract (English)

Recent continuous autoregressive TTS models operate directly on continuous speech representations, preserving rich acoustic details while leveraging the instruction-following capabilities of text LLMs. This paradigm opens new possibilities for voice cloning, instruction-controlled voice design, and speech editing, but remains susceptible to error accumulation during autoregressive generation. Existing solutions often require additional semantic modules, multi-stage tokenizer training pipelines, or complex autoregressive architectures. In this work, we propose FireRedTTS3, a simple yet effective speech generation and editing framework that mitigates error accumulation at the representation level. Specifically, we leverage a frozen Audio Encoder trained on diverse speech understanding tasks as a semantic teacher to regularize the audio feature space. This improves text-speech alignment and stabilizes autoregressive generation while keeping the overall system simple. FireRedTTS3 provides two variants: FireRedTTS3-Base for multilingual and multi-dialect zero-shot voice cloning, and FireRedTTS3-Instruct for unified voice cloning, instruction-controlled voice design, and speech editing. Experiments show that FireRedTTS3-Base achieves the best average speech intelligibility and speaker similarity among compared systems on Seed-TTS-Eval and MiniMax-MLS-Test, while FireRedTTS3-Instruct outperforms competing systems on InstructTTSEval and Ming-Freeform-Audio-Edit. These results demonstrate that semantically enriched continuous speech representations, combined with a simple architecture, enable stable, controllable, and high-fidelity speech generation and editing. Code and models are available at https://github.com/FireRedTeam/FireRedTTS3.

语音生成语音编辑大模型自回归

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。