arXiv:2510.04593eess.AScs.SD2025-10中稿 · interspeech2026被引 1

一个模型同时搞定语音识别与语音合成,还能零样本克隆声音。

UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Models

  • 用连续表示统一建模语音识别与合成,避免离散分词的信息损失。
  • 在语音识别和零样本语音合成上均达到或超过单任务顶尖水平。
  • 双注意力机制自动切换识别与合成模式,适合语音多任务研究者。

大型语言模型(LLMs)在自动语音识别(ASR)和文本到语音(TTS)系统中展现出优异性能,正逐渐成为主流方法。然而,现有方法通常分别处理这两项任务,而非采用统一框架。本文提出UniVoice,一种基于连续表示的统一LLM框架,将语音识别与语音合成无缝整合于单一模型中。该方法结合自回归建模用于语音识别,以及流匹配(flow-matching)实现高质量语音生成。为缓解自回归与流匹配模型间的固有差异,设计了双注意力机制,可动态切换因果掩码(用于识别)与双向注意力掩码(用于合成)。此外,提出的文本前缀条件语音填充方法支持高保真零样本语音克隆。实验表明,该方法在语音识别和零样本语音合成任务上均达到或超越当前单任务模型表现。本工作探索了端到端语音理解与生成的新可能。代码已开源:https://github.com/gwh22/UniVoice。

原文摘要 · Abstract (English)

Large language models (LLMs) have demonstrated promising performance in both automatic speech recognition (ASR) and text-to-speech (TTS) systems, gradually becoming the mainstream approach. However, most current approaches address these tasks separately rather than through a unified framework. This work aims to integrate these two tasks into one unified model. Although discrete speech tokenization enables joint modeling, its inherent information loss limits performance in both recognition and generation. In this work, we present UniVoice, a unified LLM framework through continuous representations that seamlessly integrates speech recognition and synthesis within a single model. Our approach combines the strengths of autoregressive modeling for speech recognition with flow matching for high-quality generation. To mitigate the inherent divergence between autoregressive and flow-matching models, we further design a dual attention mechanism, which switches between a causal mask for recognition and a bidirectional attention mask for synthesis. Furthermore, the proposed text-prefix-conditioned speech infilling method enables high-fidelity zero-shot voice cloning. Experimental results demonstrate that our method can achieve or exceed current single-task modeling methods in both ASR and zero-shot TTS tasks. This work explores new possibilities for end-to-end speech understanding and generation. Code is available at https://github.com/gwh22/UniVoice.

语音合成语音识别大模型统一框架

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。