TontaubeV1用分层编码实现低延迟语音合成，单卡即可流式输出自然语音。
TontaubeV1: Streaming Text-to-Speech with Hierarchical Codec Modeling and Bounded Context
- 采用分层双编码器结构，分离语义与声学细节，逐级优化语音质量。
- 单卡推理延迟仅200毫秒，端到端实时因子达0.08，支持长达一分钟的语音克隆。
- 适合需要低延迟、高自然度语音生成的应用场景，如对话系统与有声书。
文本转语音系统常在语音自然度与推理效率间面临权衡：更高感知质量通常伴随更高计算成本和延迟。我们提出TontaubeV1，该模型在保持自然语调的同时，可实现单张消费级显卡上的流式生成。语音通过12.5 Hz的分层双编码器（DualCodec）表示，将语义流与逐级声学细化分离。设计假设语调结构在语义流生成时已基本确立，因此分配相应资源：一个基于Qwen3-1.7B的Transformer预测语义流并决定语句时长，三个逐步缩小的Qwen3-0.6B衍生Transformer分别添加一层声学细化。文本按字符粒度分词，而非子词。共享位置的文本与音频标记支持长文本生成且上下文受限，重叠的DualCodec重建被映射至VibeVoice声学潜空间并因果解码，从而在非因果解码器下仍实现流式处理。模型支持最长一分钟的参考音频用于语音风格条件控制，主要面向英语与德语，具备多语言扩展能力。四个预测器总参数量为2.9B；在单张RTX 5090上，流式路径首帧音频延迟约200毫秒。独立非流式测试中，单输入端到端实时因子（RTF）为0.08，八并发输入聚合RTF为0.02。在基于大模型评分的有声书朗读基准测试中，TontaubeV1表现与ElevenLabs Flash v2.5相当，并优于Fish Audio S2 Pro、April 2026 Gradium API及Cartesia Sonic 3的语调自然度。模型权重已通过Tontaube社区模型许可证1.0发布于Hugging Face。
原文摘要 · Abstract (English)
Text-to-speech systems often face a trade-off between natural prosody and efficient inference: higher perceptual quality typically comes at increased computational cost and latency. We present TontaubeV1, a model that preserves natural prosody while enabling streaming from a single consumer GPU. Speech is encoded by the hierarchical DualCodec representation at 12.5 Hz, which separates a semantic stream from successive acoustic refinements. Our design assumes that prosodic structure is largely established when the semantic stream is generated, and allocates capacity accordingly: a Qwen3-1.7B-derived transformer predicts that stream and thereby the utterance duration, while three progressively smaller Qwen3-0.6B-derived transformers each add one acoustic refinement. Text is tokenized per character rather than by subword. Paired text and audio markers at shared positions support long-form generation with bounded context, and overlapping DualCodec reconstructions are mapped into the VibeVoice acoustic latent space and decoded causally, enabling streaming despite DualCodec's noncausal decoder. The model accepts up to one minute of reference audio for voice conditioning and is designed primarily for English and German, with additional multilingual support. The four predictors total 2.9B parameters; on a single RTX 5090 the streaming path reaches approximately 200 ms to first audio. In separate non-streaming measurements, the end-to-end real-time factor (RTF) is 0.08 for one input and the aggregate RTF is 0.02 across eight concurrent inputs. On our LLM-as-a-judge audiobook-reading benchmark, TontaubeV1 matches ElevenLabs Flash v2.5 and outperforms Fish Audio S2 Pro, the April 2026 Gradium API, and Cartesia Sonic 3 on prosody. The model weights are released on Hugging Face under the Tontaube Community Model License 1.0.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。