GEPARD实现低延迟语音合成,支持实时对话场景
GEPARD - Generative, Prosody-aware, Autoregressive text-to-speech model for Realtime Dialogue

- 用大模型骨干+自回归解码,文本与音频嵌入联合训练
- 单流实时因子0.067,256并发时加速达204倍
- 适配vLLM引擎,支持零样本克隆和无分类器引导
我们提出GEPARD(生成式、韵律感知、自回归语音合成模型,适用于实时对话),一种面向实时口语对话的流式语音合成模型。GEPARD采用大语言模型骨干网络,以全注意力变换器为架构,在单一解码器中联合训练文本与音频嵌入,并通过基于FSQ的神经编解码器逐块流式生成波形,随文本到达实时输出音频。核心目标是构建一个可由标准大模型引擎(vLLM)直接服务的TTS架构,不修改其计算内核。设计原则为:骨干模型使用标准全注意力变压器,所有复杂辅助机制——零样本语音克隆、文本增强、无分类器引导——均移出自回归解码循环,置于预填充阶段或直接蒸馏进权重。在流式端到端推理中,单个流实现实时因子约0.067(约15倍于实时速度);在单个服务器级GPU上,256并发时系统整体加速约204倍。本文详细阐述:(1) vLLM原生服务的系统级解决方案;(2) 自回归语音解码器的短寄存器(1-2词)失败模式,及其诊断探针与缓解方法;(3) 通过直接偏好优化(DPO)将两阶段无分类器引导的文本蒸馏为单阶段权重。
原文摘要 · Abstract (English)
We present GEPARD (Generative, Prosody-aware, Autoregressive text-to-speech model for Realtime Dialogue), a streaming text-to-speech model for real-time spoken dialogue. GEPARD generates speech autoregressively with an LLM backbone - text and audio embeddings are trained together in a single decoder-only model - and decodes it to a waveform with an FSQ-based neural codec, streaming audio chunk-by-chunk as text arrives. Our central goal is a TTS architecture served by a standard LLM engine (vLLM) without modifying its compute kernels. This defines the overarching design principle: the backbone is a standard full-attention transformer, while all non-trivial auxiliary mechanisms - zero-shot voice cloning, text augmentation, and classifier-free guidance - are moved out of the autoregressive decode loop into prefill, or distilled directly into the weights. On streaming end-to-end inference, a single stream reaches a Real-Time Factor of about 0.067 (roughly 15x faster than real-time); under 256 concurrent streams the system reaches an aggregate speedup of about 204x on a single server-class GPU. We detail: (1) system-level solutions for vLLM-native serving; (2) the "short register" (1-2 word) failure mode of autoregressive speech decoders, with diagnostic probes and a mitigation; and (3) distillation of two-pass classifier-free guidance over text into single-pass weights via Direct Preference Optimization (DPO).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。