arXiv:2605.27258cs.SDcs.AI2026-05

用20万小时开源数据训练出高性能语音合成模型,支持多语言零样本克隆。

PilotTTS: A Disciplined Modular Recipe for Competitive Speech Synthesis

论文配图:PilotTTS: A Disciplined Modular Recipe for Competitive Speech Synthesis
图 1 · 摘自论文原文
  • 轻量级架构结合跨样本配对训练,分离说话人身份与语调特征。
  • 在种子语音评估集上中文错误率仅0.87%,语音相似度达0.862,超越更大模型。
  • 适合资源有限团队复现顶尖语音合成系统,支持方言与情感合成。

构建顶级文本到语音(TTS)系统通常需要数百万小时的私有数据和复杂的多阶段架构,对资源有限的研究团队构成巨大门槛。本文提出PilotTTS,一种轻量级自回归语音合成系统,通过极简架构与严格的开源数据工程实现竞争性性能。PilotTTS仅使用20万小时数据,全部通过开源工具处理。核心贡献包括:(1) 可复现的多阶段数据处理流程,涵盖质量评估、标签标注与过滤;(2) 紧凑模型架构,采用基于Q-Former的条件机制,通过跨样本配对训练将说话人身份与语调风格解耦。统一框架下,PilotTTS支持零样本语音克隆、情绪合成(11类)、副语言特征合成(4类)及中文方言合成(14种)。在Seed-TTS Eval基准测试中,其测试英文集的词错误率(WER)为1.50%,中文集字符错误率(CER)为0.87%,语音相似度分别达0.862和0.815,优于使用更大数据集训练的系统。代码、预训练权重与完整数据处理流程已开源。

原文摘要 · Abstract (English)

Building state-of-the-art text-to-speech (TTS) systems typically demands millions of hours of proprietary data and complex multi-stage architectures, creating substantial barriers for resource-constrained research teams. In this report, we present PilotTTS, a lightweight autoregressive TTS system that achieves competitive performance through minimalist architecture and rigorous data engineering. PilotTTS is trained on only 200K hours of data processed entirely with open-source tools. Specifically, our contributions are: (1) a reproducible multi-stage data processing pipeline covering quality assessment, label annotation, and filtering, and (2) a compact model architecture that employs Q-Former-based conditioning to decouple speaker identity from speaking style via cross-sample paired training. Within a unified framework, PilotTTS supports zero-shot voice cloning, emotion synthesis (11 categories), paralinguistic synthesis (4 categories), and Chinese dialect synthesis (14 dialects). On the Seed-TTS Eval benchmark, PilotTTS achieves the lowest WER of 1.50% on test-en, a CER of 0.87% on test-zh, and the highest speaker similarity on both test sets (0.862 and 0.815), outperforming systems trained on significantly larger datasets. We release the complete data pipeline recipe, pretrained weights, and code at https://github.com/AMAPVOICE/PilotTTS.

语音合成零样本克隆开源数据轻量模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。