arXiv:2506.09349cs.CL2025-06被引 7

DrVoice用双分辨率语音表示,让语音与文本并行生成,更省算力且性能领先。

DrVoice: Parallel Speech-Text Voice Conversation Model via Dual-Resolution Speech Representations

  • 采用双分辨率语音表示,将模型输入频率从12.5Hz降至5Hz
  • 在多个语音评测集上达7B模型新SOTA,包括OpenAudioBench等
  • 适合追求高效语音生成的开发者和研究者使用

近期基于大语言模型(LLMs)的端到端(E2E)语音生成受到广泛关注,多项工作将文本型LLM扩展为生成离散语音令牌。现有方法主要分为两类:(1) 独立生成离散语音令牌,不融入LLM的自回归过程,导致文本生成对语音合成无感知;(2) 通过联合自回归建模生成交错或并行的语音-文本令牌,实现生成过程中的跨模态感知。本文提出DrVoice,一种基于联合自回归建模的并行语音-文本对话模型,引入双分辨率语音表示。值得注意的是,当前方法多使用12.5Hz输入音频表示,而本方案将输入频率降至5Hz,显著降低计算成本,并缓解语音与文本令牌间的频率差异,从而更好发挥LLMs的能力。实验表明,DrVoice-7B在OpenAudioBench、VoiceBench、UltraEval-Audio和Big Bench Audio等主流语音基准上取得7B模型新SOTA,成为领先的开源语音基础模型。

原文摘要 · Abstract (English)

Recent studies on end-to-end (E2E) speech generation with large language models (LLMs) have attracted significant community attention, with multiple works extending text-based LLMs to generate discrete speech tokens. Existing E2E approaches primarily fall into two categories: (1) Methods that generate discrete speech tokens independently without incorporating them into the LLM's autoregressive process, resulting in text generation being unaware of concurrent speech synthesis. (2) Models that generate interleaved or parallel speech-text tokens through joint autoregressive modeling, enabling mutual modality awareness during generation. This paper presents DrVoice, a parallel speech-text voice conversation model based on joint autoregressive modeling, featuring dual-resolution speech representations. Notably, while current methods utilize mainly 12.5Hz input audio representation, our proposed dual-resolution mechanism reduces the input frequency for the LLM to 5Hz, significantly reducing computational cost and alleviating the frequency discrepancy between speech and text tokens and in turn better exploiting LLMs' capabilities. Experimental results demonstrate that DrVoice-7B establishes new state-of-the-art (SOTA) on prominent speech benchmarks including OpenAudioBench, VoiceBench, UltraEval-Audio and Big Bench Audio, making it a leading open-source speech foundation model in ~7B models.

语音生成大模型并行生成双分辨率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。