arXiv:2507.09318eess.AScs.CL2025-07ACL被引 12

用流匹配实现零样本对话语音生成,速度更快更准确。

ZipVoice-Dialog: Non-Autoregressive Spoken Dialogue Generation with Flow Matching

  • 非自回归架构结合课程学习与说话人轮次嵌入
  • 6.8小时开源数据集,推理速度提升3倍以上
  • 适合语音对话系统、数字人研发人员

生成口语对话比单人语音合成更复杂,需兼顾自然的交替发言与不同说话人音色的保持。现有自回归模型存在推理延迟高、稳定性差的问题。为此,本文提出ZipVoice-Dialog,一种基于流匹配的非自回归零样本对话语音生成模型。针对直接应用流匹配导致语音可懂度低、发言切换不准的问题,提出两项有效方法:(1) 课程学习策略以确保语音与文本对齐;(2) 引入说话人-轮次嵌入控制精准发言切换。此外,还设计了支持立体声对话生成的策略。鉴于该领域缺乏训练数据,本文构建并发布了首个大规模(6.8千小时)开源口语对话数据集OpenDialog,源自真实场景语音。同时建立了评估基准,全面评测对话生成模型。实验表明,ZipVoice-Dialog在推理速度、可懂度、发言切换准确率和说话人相似性上均优于现有方法。代码、模型及数据集已公开于https://github.com/k2-fsa/ZipVoice。

原文摘要 · Abstract (English)

Generating spoken dialogue is inherently more complex than monologue text-to-speech (TTS), as it demands both realistic turn-taking and the maintenance of distinct speaker timbres. While existing autoregressive (AR) models have made progress, they often suffer from high inference latency and stability issues. To overcome these limitations, we propose ZipVoice-Dialog, a non-autoregressive (NAR) zero-shot spoken dialogue generation model based on flow-matching. Observing that applying vanilla flow-matching to dialogue generation leads to poor speech intelligibility and turn-taking precision, we introduce two simple yet effective methods to adapt flow-matching architectures for dialogue generation: (1) a curriculum learning strategy to ensure robust speech-text alignment, and (2) speaker-turn embeddings to govern precise speaker turn-taking. Additionally, we introduce dedicated strategies to support stereo dialogue generation. Recognizing the lack of training datasets in this field, we curate and release OpenDialog, the first large-scale (6.8k hours) open-source spoken dialogue dataset derived from in-the-wild speech data. Moreover, for fair and rigorous evaluations, we established a benchmark to comprehensively evaluate dialogue generation models. Experiments demonstrate the effectiveness of the proposed methods and dataset, showing that ZipVoice-Dialog achieves superior performance in inference speed, intelligibility, speaker turn-taking accuracy, and speaker similarity. Our code, model checkpoints, and the OpenDialog dataset are publicly available at https://github.com/k2-fsa/ZipVoice.

语音生成非自回归对话系统流匹配

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。