让语音与文本同步对齐,用大模型生成更准的语音。
TADA: A Generative Framework for Speech Modeling via Text-Acoustic Dual Alignment
- 通过一对一音素-文本对齐,实现语音与文本的实时同步
- 生成语音时几乎无内容幻觉,推理成本大幅降低
- 支持文本与语音混合生成,适合构建智能对话系统
现代文语转换(TTS)系统越来越多地采用大语言模型(LLM)架构,以实现可扩展、高保真、零样本语音生成。然而,这些系统通常依赖固定帧率的声学分词,导致语音序列远长于且异步于对应文本,不仅计算效率低,还常引发TTS中的幻觉现象,并加剧语音语言建模(SLM)中的模态差距。本文提出一种新型分词方案,建立连续声学特征与文本标记之间的一一对应关系,使声学与文本在大型语言模型中实现统一、单流建模。实验表明,这种同步标记在保持高保真语音重建的同时,可在潜在空间中被大语言模型有效建模,且通过融合纯文本与文本-语音模式的输出逻辑,可灵活实现文本引导的语音生成,从而弥合文本语言模型与语音能力之间的鸿沟。结果表明,该方法性能媲美顶尖TTS与SLM系统,基本消除内容幻觉,同时显著降低推理开销。
原文摘要 · Abstract (English)
Modern Text-to-Speech (TTS) systems increasingly leverage Large Language Model (LLM) architectures to achieve scalable, high-fidelity, zero-shot generation. However, these systems typically rely on fixed-frame-rate acoustic tokenization, resulting in speech sequences that are significantly longer than, and asynchronous with their corresponding text. Beyond computational inefficiency, this sequence length disparity often triggers hallucinations in TTS and amplifies the modality gap in spoken language modeling (SLM). In this paper, we propose a novel tokenization scheme that establishes one-to-one synchronization between continuous acoustic features and text tokens, enabling unified, single-stream modeling within an LLM. We demonstrate that these synchronous tokens maintain high-fidelity audio reconstruction and can be effectively modeled in a latent space by a large language model with a flow matching head. Moreover, the ability to seamlessly toggle speech modality within the context enables text-only guidance--a technique that blends logits from text-only and text-speech modes to flexibly bridge the gap toward text-only LLM intelligence. Experimental results indicate that our approach achieves performance competitive with state-of-the-art TTS and SLM systems while virtually eliminating content hallucinations and preserving linguistic integrity, all at a significantly reduced inference cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。