arXiv:2509.24650cs.SD2025-09被引 48

无需分词器的语音合成模型,实现自然流畅的语气与音色克隆。

VoxCPM: Tokenizer-Free TTS for Context-Aware Speech Generation and True-to-Life Voice Cloning

  • 分层语义声学建模,用可微量化瓶颈分离语义与细节
  • 在180万小时双语数据上训练,零样本语音合成达开源最优
  • 端到端训练,支持上下文感知的语气风格生成,适合语音应用研发

语音合成生成模型面临根本矛盾:离散符号保证稳定性但损失表现力,连续信号保留音质却因任务耦合导致误差累积。现有方法依赖预训练语音分词器,造成语义与声学的割裂,限制整体表达。本文提出无分词器的语音合成模型VoxCPM,通过半离散残差表示实现分层语义-声学建模。其架构包含可微量化瓶颈,促使文本-语义语言模型(TSLM)生成语义韵律规划,残差声学模型(RALM)恢复精细声学细节。该表示引导基于局部扩散的解码器生成高保真语音隐变量。整个系统在简单扩散目标下端到端训练,无需外部分词器。在180万小时双语语料上训练的VoxCPM-0.5B模型,在开源系统中达到零样本语音合成最佳性能,验证了该方法在表达性与稳定性上的优势。此外,模型能理解文本并推断合适语调与风格,生成具有上下文感知表达力和自然流动性的语音。为推动社区研究,VoxCPM已开放发布,许可协议为Apache 2.0。

原文摘要 · Abstract (English)

Generative models for speech synthesis face a fundamental trade-off: discrete tokens ensure stability but sacrifice expressivity, while continuous signals retain acoustic richness but suffer from error accumulation due to task entanglement. This challenge has driven the field towards multi-stage pipelines that rely on pre-trained speech tokenizers, but these create a semantic-acoustic divide, limiting holistic and expressive speech generation. We resolve these dilemma through hierarchical semantic-acoustic modeling with semi-discrete residual representations and present a novel tokenizer-free TTS model VoxCPM. Our framework introduces a differentiable quantization bottleneck that induces natural specialization: a Text-Semantic Language Model (TSLM) generates semantic-prosodic plans, while a Residual Acoustic Model (RALM) recovers fine-grained acoustic details. This hierarchical semantic-acoustic representation guides a local diffusion-based decoder to generate high-fidelity speech latents. Critically, the entire architecture is trained end-to-end under a simple diffusion objective, eliminating dependency on external speech tokenizers. Trained on a massive 1.8 million hours of bilingual corpus, our VoxCPM-0.5B model achieves state-of-the-art zero-shot TTS performance among open-source systems, demonstrating that our approach delivers expressive and stable synthesis. Besides, VoxCPM shows the capability to comprehend text to infer and generate appropriate prosody and style, delivering speech with context-aware expressiveness and natural flow. To facilitate community-driven research and development, VoxCPM is publicly accessible under Apache 2.0.

语音合成扩散模型音色克隆无分词器

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。