用能量距离建模连续潜空间语音，提升生成效率与质量
Efficient Speech Language Modeling via Energy Distance in Continuous Latent Space
- 将语音波形转为连续潜变量序列，用能量距离优化自回归建模
- 零样本和流式语音合成效果优秀，无需分层结构与量化误差
- 适合追求高效高保真语音生成的研究者与开发者
我们提出SLED，一种通过将语音波形编码为连续潜表示序列，并使用能量距离目标进行自回归建模的语音语言建模新方法。能量距离通过对比模拟样本与目标样本，提供分布差距的解析度量，实现高效训练以捕捉底层连续自回归分布。SLED摒弃了残差向量量化依赖，避免离散化误差，同时省去现有语音语言模型中常见的复杂分层架构，简化整体建模流程。在保留语音信息丰富性的同时，维持推理效率。实验证明，SLED在零样本和流式语音合成任务中均表现优异，展现出其在通用语音语言模型中的广泛应用潜力。
原文摘要 · Abstract (English)
We introduce SLED, an alternative approach to speech language modeling by encoding speech waveforms into sequences of continuous latent representations and modeling them autoregressively using an energy distance objective. The energy distance offers an analytical measure of the distributional gap by contrasting simulated and target samples, enabling efficient training to capture the underlying continuous autoregressive distribution. By bypassing reliance on residual vector quantization, SLED avoids discretization errors and eliminates the need for the complicated hierarchical architectures common in existing speech language models. It simplifies the overall modeling pipeline while preserving the richness of speech information and maintaining inference efficiency. Empirical results demonstrate that SLED achieves strong performance in both zero-shot and streaming speech synthesis, showing its potential for broader applications in general-purpose speech language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。